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REVIEWERS COMMENTS 


LARRY GOOD, Chair, Founder, and Senior Fellow, Corporation for a Skilled Workforce, 
Michigan - The Roundtable report makes a timely and enormously valuable contribution to reframing 
how we approach adult education policy in the United States. Framing the issue of basic skills 
development in Return-on-Investment terms is fundamental to building the policy and political support 
required to make a scalable impact on the largest challenge to U.S. workforce competitiveness. The 
concrete descriptions of the strategies in the six profiled states offer some great ideas for policymakers 
and practitioners at all levels to consider adopting, and as a set offer important clues about how to reframe 
basic skills development within the 2 1 st Century economic context. The reflections on the meaning of the 
PIAAC results are must reading; as the report notes, PIAAC's methodologically solid international 
comparisons paint a dire picture about risks of there being a significant portion of the U.S. workforce that 
will be left behind in a rapidly changing, skills-centered economy. The ideas from the Roundtable about 
messaging, funding, and changing practice, all grounded in an ROI framework, provide me with some 
great ideas about how we move adult education to the forefront of policy and action that is so badly 
needed. Thanks to CAAL for doing this important convening and report. 


CHERYL KING, Senior Policy Advisor, Kentucky Council on Postsecondary Education (Study 
Director, National Commission on Adult Literacy) - CAAL’s comprehensive paper on ROI is for 
everyone interested or involved in improving adult education and workforce development. In Invitation 
to a Roundtable CAAL provides an excellent overview of promising state and federal programs working 
to improve metrics and data systems. But it also provides a realistic overview about the many barriers 
impeding the sharing of data among organizations with similar goals, the lack of funding to 
address technology infrastructure needs, and the inability of some to keep up with the current realities of 
the contemporary workplace. It’s been said that we treasure what we measure, and CAAL has again 
raised the right issues at the right time about the importance of ROI and its impact on performance and 
quality. Can America compete globally when we rank below most other countries in average literacy, 
numeracy, and problem solving in technology-rich environments? Can we eradicate the skills shortage 
when only 30% of our adults have an associate or bachelor's degree? Can programs be improved if we 
don't know what's working and what isn't? Invitation to a Roundtable is a call to action. Urge your 
colleagues to read it and begin the dialogue and planning required to incorporate ROI into your 
organization. Our nation's future depends on it. 


ED MORRIS, Executive Director (Retired), Adult and Career Education, Los Angeles Unified 
School District - If California had been more engaged in serious Return-on-Investment activity a few 
years ago, the Adult Education fiasco and the destruction it caused might not have occurred. But we 
have been given a new opportunity to restore Adult Education through collaboration and careful planning 
over the next couple of years, and ROI will be an essential element of that plan and our future success. 
With Adult Education programs nationwide adjusting to new economic realities, CAAL has again 
demonstrated critical insight and leadership in fostering discussions on ROI. As evolving economic 
conditions have brought ROI to the forefront of decision-making in all financial sectors. Invitation to 
a Roundtable , along with CAAL’s earlier report. Stepping Up to ROI in Adult Education , should 
be required reading for adult educators everywhere and used to inform local discussions and 
near-term planning. 



NICOLE SMITH, Senior Economist, McCourt School of Public Policy, Georgetown University 
Center on Education and the Workforce - CAAL has shown a tremendous amount of foresight in 
putting together its panel on ROI as it relates to adult education. As we continue to recover from this 
recession, many state funding formulae are now based on a measured assessment of ROI for all social and 
economic investments. A better understanding of ROI in Adult Education puts CAAL and others in a 
better position to press for continued investment. The narrative on the value and role of education as a 
social good has slowly shifted backwards into the pipeline away from adult education, first towards 
higher ed, then K-12, and now most recently early childhood education. The flow of resources in turn has 
more and more favored younger adults and children. Though it is clear that the returns to investment are 
highest in children of pre-school age, this approach to human capital investment has created a false 
dilemma, pitting adults against children. The fact remains that the adult ed cohort will continue to need 
training to keep up with changing job requirements. Moreover, since the cohort of adult ed workers 
includes the unemployed, underemployed, disabled, returning veterans, and formerly incarcerated, robust 
estimates of ROI at all levels is a step in the right direction. 


JOHAN UVIN, Deputy Assistant Secretary, Office of Career, Technical, and Adult Education 
(formerly OVAE), U.S. Department of Education - Your November 2013 ROI Roundtable was 
effective for many reasons. One is that a national conversation about the issue of ROI for adult education 
was long overdue. CAAL should be commended for organizing this session. A second reason is the 
thought that went into putting together the group of participants. The group’s composition reflected 
expertise from various disciplines, not just adult education, all necessary to have a well-informed 
discussion, and there was research, program, policy, and practice expertise. The make-up reflected the 
cross-sectoral partnerships needed to examine how ROI is best done given the multiple subpopulations 
and contexts of adult education. The level of preparation was excellent including the background 
materials prepared by CAAL and the mini-presentations by various participants. The high quality of the 
preparation elevated the level of the discussion to where it needed to be. In addition, the discussion was 
structured in a way that encouraged the participants to expand or create knowledge together progressing 
from building shared understanding of constructs and moving toward specific opportunities and possible 
solutions. The resulting report. Invitation to a Roundtable , is built on the same inclusive spirit that 
marked the Roundtable. It offers useful suggestions on how to proceed and should open the door to other 
good thinking about this very important topic. 



INVITATION TO A ROUNDTABLE: 

A Discussion of Return-on-Investment in Adult Education 


A. INTRODUCTION 

In September 2013, based on a national survey of state ABE direetors or designates, CAAL 
published STEPPING UP TO ROI IN ADULT EDUCATION: A Survey of State Activity} 

This paper reported on the states’ aetivity, from the perspective of the respondents, in collecting 
and using retum-on-investment data in their workforce adult education programs. It examined 
six states in detail (AR, CT, KY, NJ, OR, and VA), and proposed several next-step actions for 
government, research, business, and philanthropy in building future state capacity to meet the 
ROI challenge. 

As states engage in more comprehensive planning, form local and state coalitions to bring more 
groups and interests to the planning table, and organize to provide a wider array of effective 
workforce and college readiness services, state ABE directors are in the midst of a seismic shift 
in Adult Education, as they are increasingly called upon to provide an array of new services. 
More and more, they are one of many teams of interest engaged in statewide planning for Adult 
Education. To tap into a wider range of interests, CAAL decided to build on its survey by 
bringing together an invitational Roundtable of other kinds of national and state leaders, which 
we did on November 8, 2013 in New York City. Our goal was to provide a richer range of 
insights and strategies to help states build ROI capacity in Adult Education. 

We designed the day around a few topical panels. One focused on current ROI updates and 
issues in six leadership states (AR, CA, IN, KY, MN, and VA). A representative of the much- 
heralded Arkansas WAGE program was invited but was unable at the last minute to attend; he 
nevertheless provided ROI data. Another panel considered the challenges in developing ROI 
evidence for special needs subgroups (ESL, Immigrants, Corrections, the Working Poor, 
and Family Literacy). A third presented key findings from PIAAC, so that the participants 
could begin to consider how PIAAC data might help add substance and direction to state ROI 
plarming and data collection efforts. A fourth considered the relevance of ROI efforts in Canada 
for the U.S. 

To conclude the day, the participants were asked for next-step suggestions in several areas: 
research; strategic activities for state and federal government; projects or measures to help 
states factor special-needs groups into their services and ROI activity; and steps that can/should 
be taken by philanthropy, business, unions, and community colleges to foster development 
of state ROI activity in Adult Education. They were urged to give special consideration to 
low-skilled adults. 

This paper, INVITATION TO A ROUNDTABLE: Strategies and Insights to Help Build 
ROI Capacity, reports on the CAAL Roundtable. It does so in a conversational format adapted 


' STEPPING UP TO ROI IN ADULT EDUCATION, by James Parker and Gail Spangenberg, is available from the 
CAAL website at http://www.caalusa.org/Ster)t)ingUptoROI.r)df . 
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from the transcribed conversation that invites you, the reader, to the table. It should be read as a 
companion paper to STEPPING UP. The main body of the paper presents elements of the 
various discussion strands, often paraphrasing what the participants said, and it synthesizes the 
next-step suggestions and ideas (beginning on p. 19) generated by the discussion. Appendix I 
provides a brief biography of the authors. Appendix II lists the Roundtable participants. 
Appendix III provides a few resources to supplement the comprehensive resource listing 
provided in our STEPPING UP report. Appendix IV contains informational handouts developed 
for the ROI Roundtable regarding PIAAC and the Arkansas WAGE program. 

The Annie E. Casey and Charles Stewart Mott Foundations provided grant funding for the ROI 
Roundtable and this report, and McGraw Hill Financial provided extensive in-kind support. 

B. SOME ROI CAUTIONS 

“Why is it important to bother about ROI?” CAAL asked the group at the outset. “Because,” 
said the U.S. Department of Education representative, “we know we are funding things that 
work, but we also know we’re funding things that don’t work. We need to be able to say that if a 
person participates in this program or that, you can count on certain types of economic or social 
or service returns.” 

But determining ROI is not a simple matter. The CAAL survey showed ROI to be both mixed in 
definition and in a state of flux. For example, at the time of the survey, according to ABE state 
directors and designates, 26 states were measuring one or more of the workforce education 
indicators for Adult Education proposed in the still-to-be-enacted Workforce Investment Act. 
Eleven states tracked workforce readiness certificates as a program outcome or said they soon 
would be. Some 21 states were able to measure at least one or some of the general program 
benefits/outcomes (such as GED acquisition) thought to be timely by studies of CAAL, the 
Department of Education, and others. Some 22 states were giving serious attention to trying to 
measure the use of new technology on the job. Only a few states were doing nothing at all. 

Add to this mix certain other variables, and the challenges of ROI are obvious. For example, 
one participant noted that to develop ROI data solely on the measures CAAL surveyed in 
STEPPING UP, state data systems will have to be pushed even more than they are now. “ABE, 
postsecondary, and even workforce groups will have to reform their systems and upgrade them 
so that they can track students into not only postsecondary education but can do very significant 
wage record matches.” Developing data systems and engaging in well-targeted ROI is going to 
“be critical over the next couple of years,” he said, and hopefully it will “bring a higher profile to 
adult education and in a favorable way.” 

Another participant asserted that ROI is more complex than CAAL’s work suggests because part 
of what we need to be doing is tracking costs. She pointed out that tracking costs requires more 
than just looking at Adult Education state budgets because programs must increasingly leverage 
resources from a number of different other agencies as well as government, foundations, and the 
business community. It isn’t very helpful, she said, to “track and talk about benefits without also 
talking about costs.” And to complicate the matter, we don’t always know whom or what 
program to credit for an outcome. For example, who would be credited if a student was both a 
TANF recipient and an enrollee in adult education and/or college? 
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Moreover, ROI, in the view of several participants, is not just about employment or college 
readiness gains or workforce skills programming, and we should not lose sight of that. There are 
many other kinds of valuable Adult Education activities, both formal and informal, and states 
need to document evidence on successes and impacts in those program areas, too. 

And finally, as we strive to develop better ROI state capacity so as to make a solid investment 
case, we should keep in mind other societal costs that drain state budgets. For instance, high 
school drop-outs are two times as likely to be unemployed, three times as likely to be in poverty, 
and eight times as likely to be incarcerated. Facts like this are strong selling points and critically 
important to effective messaging. 

C. PACESETTERS: ROI Unfolding in Six States 

Even though the states have a considerable distance to go on the ROI front overall, most are 
paying greater attention than just a few years ago to ROI definitions, program evaluation, 
outcomes, and impact in workforce/workplace adult education. 

According to CAAF’s survey, two-fifths of the states have invested time, effort, and expense in 
special initiatives over the past few years. Many states can list specific outcomes of their 
workforce programs to increase worker skills and performance in the workplace and some can 
provide quantifiable data about those outcomes. Most understand the limitations of the National 
Reporting System, and most feel that NRS performance data does not tell the complete story of 
their actual program accomplishments and range of services. Some await and hope for new 
legislative requirements or guidelines to support ROI-type data systems and outcome 
measurements they would like to undertake. The best efforts are based on partnerships with a 
variety of interest groups and extensive cooperation with business, as well as an understanding of 
Adult Education’s changing service role. 

Six leadership states are discussed below. Kentucky, Virginia, and Arkansas were profiled in 
depth in STEPPING UP. (Note: Although last-minute difficulties prevented Arkansas’ 
representative from joining the Roundtable, its adult education and workforce skills effort is 
among the most highly developed in the nation. It is discussed below in terms of data submitted 
to CAAL for its Roundtable presentation.) 

1. California 

Over the past five years, as widely publicized, California lost large portions of its state Adult 
Education funding. That funding, including WIA, shrank from about $753 million annually to 
about $200 million. However, thanks in part to leadership from the community college system 
and a savvy and committed governor. Adult Education has recently been given the opportunity to 
“plan itself back into existence.’’ $25 million has been set-aside over the next two years for local 
groups to plan and apply for comprehensive programs covering elementary, basic skills, high 
school diploma, immigrant and ESL programs, and career-technical education. The Department 
of Education and the California Community College Chancellor’s Office have been charged to 
collaborate to develop, coordinate, and support these changes. 
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In this new initiative, local consortia must include provisions in their plans to accelerate a 
student’s progress— such as modularization, career pathways, contextualized basic skills, 
orientation programs, and bridge programs. They must also include research-based program 
strategies that accelerate students’ learning and move these adults quickly through a pipeline. 

The development of metrics to determine ROI and the impact of ROI on these programs will be a 
requirement. The focus of services, to be organized through regional consortia, are to be 
elementary and basic skills, classes required for high school diploma or equivalency, services for 
immigrants and ESL, and programs for disabled adults. Outcomes to be tracked are certificate 
attainment, persistence in classes, and readiness for job training and college, all according to 
“research-based metrics.” And, of equal significance, the plans must include new ways to 
advance professional development. 

A website has been set up for the new California initiative to keep people informed about 
progress, issues, and activities as the future planning goes forward.^ ROI evidence clearly will 
be an increasingly essential part of the overall California effort. “Plan and plan well,” initiative 
leaders have been charged, and “come to us in 2015 with evidence that you have.” 

2. Indiana 

Under leadership from the State Chamber of Commerce, Indiana is well advanced in its adult 
education and workforce skills programming and its use of ROI data. The adult education 
structure there has been substantially reformed, a goal achieved in part by an in-depth 
partnership with the business community and a shift of governance from the Department of 
Education to the Department of Workforce Development. Our Roundtable participant gave 
heavy emphasis to the importance, not just in Indiana, but around the country, of developing an 
approach that taps into the various funding streams at the disposal of Adult Education today. 

But, she said, “to think about that and to do it in a new way takes a lot of work because you have 
to really press the flexibilities of those funding streams, and you must be more client-centered 
than program-centered. You also have to move toward funding outcomes instead of inputs, and 
that takes a lot of work, too. You must track these things very carefully to continually make the 
case for funding.” 

The state’s work is closely tied to ROI thinking. Its plan includes three key provisions: (a) an 
outcome-based performance funding structure; (b) a new data system that tracks everything 
much more carefully and comparatively among programs, and (c) a system where data is shared 
by three key agencies— the Department of Education, Department of Workforce Development, 
and Commission for Higher Education. The goal of collaboration is to cormect education to 
wage and employment experience. 

Bridge programming is at the core of everything. Under this rubric, Indiana tracks career 
pathways of possible relevance to adult students. It also tracks industry-recognized credentials 
earned while enrolled in adult education and subsequent to getting a GED/equivalent, student 


^ California’s new website is at http://ab86.cccco.edu . 

^ For detailed information about the Indiana initiative, see http://www.readvindiana.org . Also see 
http://www.insideindianabusiness.com/contributors.asp?id=1129 . 


4 


completion, attainment of actual credentials, students’ entry into employment in the occupation 
for which they studied, and wage data changes over time. The Indiana Business Research Center 
and Ball State University have been conducting comparative analyses of the data collected. 

On a different track, the Indiana Commission for Higher Education recently embarked on its 
second ROI report for the postsecondary system. It is drawing on data from the three key 
agencies cited above. In addition, a new Career Council was recently created by legislative 
statute to oversee the state’s adult education and workforce development effort - including 
attention to the functioning and further development of its longitudinal data system. 

It is interesting to note that on the basis of well-defined learning outcomes and an excellent data 
system, the state of Indiana has apparently “gone from fifth to third in the country in terms of 
sheer numbers of credentialed manufacturing workers.” But one of the Roundtable participants 
pointed out that “almost all states are now getting federal resources for their longitudinal data 
systems and the focus and insistence on incorporating adult education into that system is not as 
strong as it should be. ” She stressed that “every state should be paying attention to that because 
having adult education as part of the total system is critical. ” 

3. Kentucky 

For the Kentucky representative, the most important element of service planning provision and 
of determining ROI is having a “clear, concise, consistent, comprehensive message that’s 
compelling.” It is important to develop a context using good data and have a call to action that 
everyone understands. In his state, the context, or message, is shaped largely by an 
understanding of the relationship between educating workers and potential workers and state 
economic development. Good ROI evidence is vital. Our participant said that everyone is being 
pushed to do more with less and less funding these days, and he noted that unless we plan well in 
a clear context and can show specific outcomes we could find ourselves doing less and less 
because we will have “nothing at all” in terms of funding. 

The analysis shows that Adult Education must help prepare adults for postsecondary education. 
The data also shows compellingly that “the more you learn, the more you earn.” Kentucky takes 
these propositions seriously as it tracks retum-on- investment. To give one example, about 20 
percent of all school graduates in the state hold the GED or an alternate diploma. On average, 
these graduates earn about $9,400 more a year than a non-graduate in the state. And the potential 
increased earnings for GED graduates in just 2012-13 will be nearly $2.5 billion over a 30-year 
career, a cohort whose unemployment rates will also be substantially lower."^ This kind of 
analysis and outcome data is an ongoing part of messaging in Kentucky. 

Kentucky has managed to reduce its working age unemployed population from 21% only a short 
time ago down to 15% at the present time. In the process, the state has moved ahead of six other 
states on this measure and they benchmark their program against those other states. Moreover, 


For this and other KY ROI information and analysis, see http://www.kvae.kv.gov/NR/rdonlvres/743D9794-89AC- 
4EB 1 -9D96- 1 20766653 85B/0/KYAEProfde.pdf . 
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Adult Education has been established as a department level ageney, an organizational change 
thought to be partly responsible for this suecess. Adult edueation is planned at a high poliey 
level, and its department level status enables it to work with a wider range of key partners aeross 
the state. Adult edueation leaders meet regularly with the Department of Workforce Investment, 
the ehancellor of the community college system, and the business community, among others. 

Many of Kentueky’s 120 eounties are not loeated near a community or technical college. Thus, 
it is seen as important for Adult Education to help individuals in those areas obtain high sehool 
equivalency diplomas, national career rating eertificates, and some level of employability skills, 
and then to eonnect these with oceupational skills training. 

Beeause it believes messaging to be so important, Kentucky is looking at branding its ROI 
results with a Skill Up Kentucky label, whieh it sees a way to engage people and get them to 
understand the importance of workforce skills, and college and job preparedness. “If you 
look at how state budgets are cut up,” said our participant, we “educate, medieate, and 
ineareerate, and too little goes to educate. We must stop the pipeline leaks, and be sure we 
reeruit and retain adult edueation students and move them to needed outcomes. To do this, we 
must remember that it’s about the students. We should think not only about messaging but about 
how to re-engineer the adult education system to make it responsive to eontemporary needs.” 

4. Minnesota 

In Minnesota, Adult Education leaders believe that the single largest ROI variable of interest at 
the federal level is the size of the state’s commitment. So they are eonsidering how to enhance 
federal NRS data by reporting outcomes per eapita or ineentivizing an inereased state match with 
increased flexibility for how federal funds are spent. This is considered a priority issue because 
“it is the federal law that drives ABE programming in Minnesota, and even though the state 
provides more dollar for dollar than the federal government does, the federal law is mueh better 
than our state ABE law. We need it and its funding to be solidly in plaee.” 

In addition, together with the governor’s workforee development council (the state WIB), 
Minnesota has implemented a “proof of concept” ROI service model that ineludes benefit 
categories and eost eategories. The benefit categories are ehange in earnings and fringes, ehange 
in taxes, and ehange in public benefits such as TANF and SNAP reductions. 

Overall, the Minnesota ROI effort currently includes data from the Adult Edueation system, the 
Department of Labor, and SLEDS^ (the Statewide Longitudinal Education Data System). The 
state is presently trying to merge these data systems into a single state longitudinal system. They 
indieate that they have been helped in this regard by their participation in the Joyce Foundation’s 
Shifting Gears program.^ 

The work is also being supplemented by other data gathering activity. To give one example. 
Adult Edueation has established a set of “direct data sharing agreements” between adult basic 


^ The SLEDS website can be accessed at http://mnp20.org/working groups/longitudinal data svstem.html 
® For information about Shifting Gears, http://www.iovcefdn.org/shifting-gears . 
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education, the workforce system, the human resource system, and higher education, so that data 
can be tracked over time in a more complete and useful way. To give another, Minnesota has a 
large immigrant and refugee population. They cannot presently be isolated as a subgroup within 
the current data system, so that issue is being addressed collaboratively with the Department of 
Human Services. 

5. Virginia 

Virginia’s workforce education program, Plugged In VA^, was started with innovation funds 
from the governor in 2008. The first program, an intense six-month program for low-skilled 
adults testing at a 9* *-grade equivalency level, was rolled out in a single rural location in 2009. It 
is now operating in 13 locations which are expected to grow to 39 by the end of the 2014 fiscal 
year. Because of its focus and early achievements, the program has apparently become a guiding 
force for adult education in the state as a whole. Partnering primarily with local community 
colleges and prospective employers, the goal is to move these low-skilled adults to college 
readiness and jobs in high demand, high- wage industries. Programming is linked to the results 
of labor market research in different regions of the state. 

Based on achievements demonstrated through ROI data and good messaging, Plugged In Va has 
been able to attract new funding from local industries and the local WIB, and by 2010 the 
program was already doing so well that it achieved a line item in the governor’s budget. 

Virginia was a participant in the Policy to Performance Project of the U.S. Department of 
Education. According to our Virginia participant, this experience was highly valuable because 
it provided the opportunity to bring in new kinds of partners. Adult Education is working with 
local WIBs and businesses, community colleges, and various departments and agencies of 
state government including Social Services. It also works with the Virginia Manufacturing 
Association, the Board of Veterans Services, and the Virginia Employment Commission (VEC 
brings in Department of Labor funding). 

The curriculum is customized to fit the specific local context, and internships, career readiness 
certificates (geared to Work Keys), and digital literacy are features of the program. Career 
readiness certificates are offered at bronze, silver, and gold achievement levels, which are 
recognized by employers. The state is now piloting badge systems, awarding badges based on 

O 

skills and expertise that industry partners want to see. Plugged In VA is also attracting attention 
from other states. For example, Kentucky is looking at the model as a way to extend its own 
outreach and services. 

Plugged In Va focuses on low-skilled adults at upper levels of proficiency, 9‘*' grade equivalency 
and above. But the state is sensitive to the needs of much lower-skilled adults and is trying to 
address those needs through several pilot programs, a few of which are considered quite 
successful. Virgina is committed to serving hard-to-reach populations, including those with ESL 


’ http://www.pluggedinva.com 

* See http://lincs.ed.gov/publications/pdf/AIR Digital Badge Report 508.pdf . which explores the potential of 
badges for use in adult education programs. 
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needs. The state’s data system apparently does a very good job of tracking and evaluating every 
student who goes through the Plugged In Va program, but it does not yet track them well after 
they move into other programs. A pilot program is in place to correct this limitation. 

6. Arkansas 

Arkansas’ adult education program is one of the most comprehensively planned and evaluated in 
the country.^ It is carried out substantially in cooperation with business and industry. Among its 
instructional strands, as in many states, are GED preparation, basic ABE classes for adults 
functioning below 12‘*' grade level, ESL instruction, and programs to lift basic skills to enhance 
employability (through its Career Readiness Certificate and WAGE programming). 

The state has a solid data gathering operation, which enables it to determine outcomes and 
impacts of all kinds, including detailed economic benefits from adult education services. 

For example, according to information supplied to the CAAL Roundtable, the state’s return on 
investment from its GED funding in 2008-2009 netted a rate of return on investment of 43% or 
nearly $27 million over and above the $18 million investment also recovered over the 10-year 
working career of the GED recipient. “Similar results apply to other years,” the state 
Association of Continuing and Adult Education says. In addition, lifetime total health savings 
for high school graduates in Arkansas is estimated to be about $40,500 per person compared to a 
non-graduate, or some $211 million over a working career for the 2008-2009 cohort. 

It should be noted that the WAGE program, which awards hundreds of certificates each year, is 
given free to business and industry partners, incumbent workers, and adults seeking employment. 
It has developed several dozen competencies for its students in four areas: Communications, 
Math, Reading, and Writing. Participants can enroll at any time and exit when their individual 
goals are completed. (See Appendix IV for a listing of the WAGE competencies and additional 
outcome data.) 

D. A PARKING LOT ISSUE: Using Social Security Numbers 

During discussion of the Minnesota ROI and planning experience, there was a lively exchange 
among the participants about the varied use of social security numbers across the country for 
gathering and matching data. There is considerable variety among the states, including those 
present at the Roundtable: 

• Minnesota has statutory authority to ask for social security numbers but does not 

require them. It is able to get social security numbers for about 80 percent of its program 
participants. The state gets a good deal of useful data in this way, although the data 
system was characterized as “a fuzzy matching engine” meaning that the state is not 
always precisely sure what the data represents, a problem they are working on now. 


® For detailed information: http://ace.arkansas.gov/adultEducation/Pages/default.aspx . 
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• Kentucky uses social security data and in some cases (usually with ESL adults) issues a 
unique student ID number for tracking/matching. 

• In California, some community colleges collect social security numbers, and some 
important data matching starts with that. Where there are no social security numbers, 
first and last name, date of birth, and zip code information is used by the community 
college system. The K-12 system uses social security numbers to match for GED, but 
they do not do wage or unemployment insurance data matching. 

• The Chamber-driven effort in Indiana involves different agencies that use different 
identifying numbers for data analysis purposes.'® For example, for workforce purposes it 
uses social security numbers. For adult education it uses student test numbers (STN), and 
for higher education it uses an assigned unique number. To match across differences, it 
takes an algorythmic approach to connect the various data points. 

• Virginia does not require social security numbers for adults who receive services through 
its state-funded programs; however, social security numbers are the only way Plugged In 
Va can match data with the Virginia Employment Commission (VEC), so learners are 
encouraged to provide them at intake. A pilot now in process with the state’s longitudinal 
data system is expected to eliminate the need to use social security numbers. 

One of the national participants pointed out that a number of states are prohibited by law from 
collecting social security numbers. He said that about 21 states have legal or other issues with 
using them as a route to data matching. However, these states and some national groups that 
work with states have found other ways to get the data they need. For instance, the highly 
ambitious and accomplished national credentialing program of the Manufacturing Institute of the 
National Association of Manufacturers gets close to 90% of what they need without the use of 
social security numbers by matching its program data with public sector data." 

A final set of comments on the need to use social security numbers has to do with immigration 
reform. It was noted that we should keep in mind that social security numbers are a key way to 
verify continuous employment, which “is going to be really important for the undocumented and 
for many youth and adults as well.” The discussants agreed that the question of how and 
whether to use social security generated data needs to stay on the agenda as a “parking lot issue.” 

E. ROI & SPECIAL-NEEDS POPULATIONS 

The purpose of this strand of the discussion was to explore issues that may be unique to state 
planning and ROI activity for certain “special needs” groups: ESL/immigrants, the corrections 
population, the working poor, and family literacy. 


For detailed information on how the Indiana data gathering and accountability system works, go to 
http://www.doe.in.gov/accountabilitv/data-collection . 

" For details about the Manufacturing Institute, its bold and successful Certificate program, and its approach to 
determining outcomes, see http://www.themanufacturinginstitute.org/About/About-Us.aspx . 
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1. ESL/Immigrants 


The ESL representative pointed out that up to half of the students in the adult edueation system 
are immigrants at the beginning ESL level. Yet beyond the NRS, she said, “we don’t have any 
data that shows what differenee we’re making with these folks.” We know in general that these 
students “learn a little bit more English and tend to move up one level, usually not more than 
that.” This is not nearly enough for the ESL student to “make significant life changes in terms of 
economic gain, quality of life, and moving into jobs that pay family- sustaining wages.” We need 
to focus in our planning and ROI activity on a more rigorous education for this sub-group and 
“to help people understand that it is worthwhile to persevere.” As we advance understanding of 
ROI, there’s not much point, she stressed, in collecting outcome data that’s geared to a system 
that is not really responsive to actual service needs. “There’s a strong need to have ESL appear in 
the longitudinal studies that the states collect. And one of the things we need to know going 
forward is which adult education students started in an ESL program as they move up through 
levels of learning.” 

It was observed that we have a tendency in adult education service provision to lump all ESL 
students into the same category when it comes to programming, regardless of their proficiency 
level. Programs need to be designed differently for low-skilled adults with ESL needs than they 
are for well-educated adults who just happen to need to learn English. The discussants felt that 
we should be building our system to take that difference into account, and track data accordingly. 

Another potential problem has to do with creaming. In current reauthorization bills, a low- 
skilled immigrant or refugee is unlikely to be served with WIA dollars, given the outcomes that 
would be required. So there is a danger that the creaming we already do in adult education and 
workforce skills development will get worse. We need to be diligent in protecting against that. 

On the immigration front generally, it was observed that as much as 50% of low- wage workers 
in the U.S. are foreign bom, and that we haven’t even begun to make the investment needed to 
provide the adult ESL services they need. It is as though “having low wage workers is 
something we mean to do.” Up to now, we have had the hope that the children of these people 
would have better success, but “we are beginning to see that the children are not succeeding.” 
That’s what the pre-K initiative is all about. We need a sort of Even Start program back, 
someone said, but not call it that, and we need a substantial investment in services for low- 
literate ESL parents as well and pre-K services, which should be a key element in thinking 
about ROI. 

The Roundtable discussion was organized largely on economic return on investment, looking 
mostly at programs that integrate workforce development with adult basic education and ESL. 
But a number of participants stressed that there are other big social and political goals for 
students and programs, goals that focus on the role of community member and family member as 
well as worker. ROI also needs to take account of these outcomes. 


Federal Adult Education: A Legislative History 1964-2013, recently published by the U.S. Department of 
Education, indicates that between 2001-2011 ESL students comprised an average of 44% (47% in 2006-07) of Adult 
Education (WIA Title II) emollments (although WIA appropriations accounted for only about 12% of Adult 
Education funding). The report can be accessed at 
https://lincs.ed.gov/publications/pdf/Adult Ed History Renort.ndf . 
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To reinforce this point, the participant from New York City spoke about ROI in connection with 
an anti-deportation program for undocumented young adult immigrants in the City. He said the 
New York City Council has made an 18 million dollar investment in providing high school 
equivalency seats in the program. The program participants will hopefully be eligible for 
“deferred action” on deportation once they meet certain criteria. At this point, it is apparently 
not clear to the program staff precisely what outcomes should be measured. He posed these 
questions: Isn’t re-engagement with the adult education system an ROI outcome in itself? Isn’t 
it an outcome when a person gains the ability to help elderly parents navigate the system of 
services available to them? Isn’t it also an outcome when an undocumented immigrant gains the 
ability to access legal services? 

2. Corrections 

OCTAE (Office of Career, Technical, and Adult Education, formerly OVAE) has a major new 
commitment to correctional education, a critical area of unmet need in Adult Education. As a 

1 O 

result of powerful research findings from a 2013 Rand Corporation report and work by the 
Vera Institute of Justice'"^ and others, OCTAE is presently testing a new model approach in re- 
entry education through a variety of demonstration projects. OCTAE sees an enormous ROI 
opportunity because recidivism is what most people who work at all levels of the prison system 
seem to care about. 

The demonstration projects are being carried out through a partnership between OCTAE and the 
Department of Justice. There is strong ROI data to support their work, and further ROI results 
will be generated by the demonstrations. 

The direct costs of confinement are readily available and widely known to be huge. Depending 
on the level of security involved in confinement, “the annual cost may range anywhere from 
$35,000 to $95,000 per person, and for that amount you can actually send a person to 
Harvard. . .” The indirect costs are even larger, so we’re only touching the tip of the iceberg. By 
contrast to these enormous direct and indirect costs, the direct cost of providing the education 
considered in the Rand study was somewhere between $1,400 and $1,700 per person. 

Moreover, Rand’s study concludes that participants in correctional education have “43 percent 
lower odds of recidivating” than nonparticipants. And the evidence shows them to have a 
substantially higher likelihood of post-release employment (13%). The Rand findings make a 
powerful case for education participation and intervention. 

To some extent, that report shows the effects of correctional education on the odds of 
recidivating according to program type: adult basic, secondary, vocational, and postsecondary 
education. However, analysts believe that more research is needed to firmly show cause-and- 


The Rand Corporation study was the result of an award given under a mandated competition hy the Second 
Chance Act of 2007. It is titled Evaluating the Effectiveness of Correctional Education: A Meta-Analysis of 
Programs that Provide Education to Incarcerated Adults, by Lois Davis, Robert Bozick, et al, and available 
from http://www.rand.org/pubs/research reports/RR266.html#abstract . Also see the slide-set used for Corrections 
in CAAL’s Nov. 8* ROI Roundtable, at http://www.caalusa.org/DeptEdCorrections&Reentrv.pdf . 

http://www.vera.org/centers/substance-use-and-mental-health-program . 
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effect relationships between recidivism and program type. It is interesting to note that inmates 
exposed to computer-assisted instruction learned slightly more in reading and substantially more 
in math in the same amount of instructional time. 

A key goal of OCTAE’S corrections work is to reduce the risk to government investments in 
corrections education by focusing on proven programs and “doing all the ROI work before 
program implementation.” A program operating at NYC’s Riker’s Island was cited as an 
exemplary model. The program uses the specific outcome measure of “number of bed days,” 
and the prison knows that if they can reduce bed days by a predetermined number they can 
actually shut down a wing and achieve substantial savings. Because planners of the Riker’s 
program did so much advance analysis and knew what measure(s) were suitable for their context, 
they were able to attract funding from third party resources, such as Goldman Sachs and the 
Bloomberg Foundation. 

3. The Working Poor 

Even though the “working poor” category cuts across other many groups of low-skilled adults in 
need of service, CAAL felt a strand focused on it would add an important element to the ROI 
Roundtable discussion. The participant on this topic directs The Working Poor Family Project 
(WPFP), a 12-year old national initiative supported by the Casey, Ford, Kresge, and Joyce 
Foundations. The project’s purpose is to strengthen state policies on behalf of low-income 
working families. Its work is carried out in 23 states, primarily through support to nonprofit 
organizations that do the policy work. WPFP is motivated by a belief that the ability of low- 
income working parents to move up in jobs and achieve economic success depends on their 
access to and successful participation in adult education and higher education programs. 

WPFP plans and evaluates its activity on the basis of data drawn annually from the Population 
Reference Bureau on the number and percent of low- income working families at the national and 
state levels. This data shows that one of three working families in America is low-income, up 
from one in four 12 years ago. A key characteristic of low-income working families is that large 
numbers of them do not have the education or skills needed to get better jobs. For example, 

30 percent of these families have a parent without a high school diploma or equivalent, and 50 
percent have a parent with no experience at any level of postsecondary education. Many of 
these people are working very hard in the jobs they have but they are “falling short.” Thus, as 
WPFP works with the states, it gives heavy attention to moving family members into ABE and 
college education. 

Messaging was underscored as having very high importance for the working poor. The way we 
speak about and frame the need and the outcomes for the working poor should reflect that these 
people want to work, to participate, and to do better. Decision-makers are sometimes skeptical 
about that. In particular, our message and ROI framework should make it clear to state 
legislatures and state government that not only are these deserving people, but that the odds are 
often stacked against them. We need to pay better attention to how interventions are structured, 
to be sure they are really designed to serve this working population. 

This means developing connections to social services, and realizing that we have to think about 
adults out there working now as well as young adults coming out of the K-12 system. And then 
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we need to think about ROI in terms of the benefits of specific strategies. We shouldn’t be 
thinking only about cost effectiveness, but also about how well we achieve the specific program 
outcomes we set for those we serve. Put another way, we should have a solid understanding 
of how effective our strategies are in reaching predetermined goals before we try to do cost 
benefit analysis. 

In the United States, it was said, skills pay off more than in almost any other country in the 
world, and having low skills dramatically increases our problems of inequality and low earnings. 
We know that adults in low-skilled occupations face a higher risk of losing even elementary 
skills because they’re not using them in their jobs. So as we develop services and strategies, we 
should be aware that more highly- skilled occupations tend to provide more opportunities for 
using, maintaining, and improving skills. We shouldn’t be preparing the low-skilled working 
poor to compete with people holding low- wage jobs. 

In short, our panelist said, even though we are not doing a very good job of determining ROI for 
the working poor, positive things are happening to move us in the right direction. For example, 
OCTAE is currently working to develop strategies that pertain to the needs of the working poor. 
They are not focusing on the working poor per se, and that emphasis is very encouraging. 
OCTAE is also putting real energy into building the state longitudinal data systems that are so 
important to everything. Moreover, it is clear from CAAL’s survey and this discussion that a 
number of states are really rethinking the structure and purpose of adult education and workforce 
skills development. That’s quite a lot to build on. 

4. Family Literacy 

Family literacy is not a “special-needs” group in the same sense that the other three subgroups 
are. But it should not be overlooked in discussions of ROI because it is an essential component 
of the Adult Education system. Because it is one of the leading players in family literacy, 
the Barbara Bush Foundation for Family Literacy was represented at the Roundtable. It has 
been engaged in ROI activity in a major way and provides another window into the collective 
ROI experience. 

The Foundation celebrates its 25th anniversary this year. It decided a couple of years ago that it 
should know more about the outcomes of the programs it funds and launched an ongoing data 
gathering effort involving a wide array of its funded projects. Depending on program type and 
purpose, it began to collect data on such variables as the following: grade-level improvements, 
set-up costs, operation and maintenance costs, sustainability, impact on family literacy levels, 
impact on student vocabulary and listening comprehension, retention rates, and parental 
engagement. Heavy emphasis is being given to the degree to which a program is self-sustaining 
after five years, and to its scalability (how replicable it is). 

The data gathered in the effort is being fed into a “dashboard” that creates graphs so that staff 
can easily evaluate the results. The Foundation has already undergone some restructuring and 
set new policy directions, and it is beginning to focus its grant-giving on certain models and 
program approaches. 
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For example, in September 2014, the Foundation will open its first eharter school wrap-around 
family literacy program in Detroit in cooperation with the YMCA and an already-large charter 
school provider in the city. It will also develop its charter school model in a few other states. 
Moreover, as part of its future mission, it will “focus more on parents than we usually do.” It 
was noted that conversations about early childhood development too often overlook the fact that 
“it’s not the child who determines the poverty level of the household, but the parent.” The 
Foundation recognizes that parents need to be enabled to hold high-wage jobs and earn a family- 
sustaining wage. 

The Foundation’s overall goal in its data gathering is to “enable the staff to determine if the 
Foundation’s spending gets a good return on investment.” The Foundation also considers the 
impact a funded program’s experience will have on its own cash flow. 

F. PIAAC AS IT RELATES TO ROI & STATE PLANNING 

PIAAC is the largest, most innovative, most complex study of adult skills ever undertaken. As 
such, it is a very rich data set that adult education leaders need to be aware of and that should be 
factored into planning and policymaking at every level. For that reason, CAAL included a major 
strand on PIAAC in the ROI Roundtable, for the benefit of participants who were not yet 
familiar with the findings. A surprising number were not. 

Appendix IV (pp. 35 ff.) gives the specific PIAAC findings presented at the Roundtable — 
through an array of charts and figures adapted from the extensive PIAAC databases operated by 
the National Center for Educational Statistics, the American Institutes for Research, and the 
Organization for Economic Cooperation and Development (OECD).'^ 

I. The Key Is Adaptability 

Everyone understands that the forces of globalization and advances in technology have changed 
the world dramatically almost overnight. But everyone does not necessarily understand the 
consequences of that change for individuals and societies in terms of the increased need for 
education and solid basic skills. The game has changed from when many of us in Adult 
Education were young. Now there are a whole different set of rules. It’s important to understand 
this when thinking about the PIAAC data. “It’s not necessarily the strongest or the most 
intelligent among us who will determine our future, but the ones that are most adaptable.” 

Our success in adapting to the changes at home and abroad will very likely “determine both the 
rate and shape of our economic growth and decisions we make about how to share that growth.” 
Unfortunately, “although America’s productivity has continued to grow in recent decades after 
World War II. . .we just don’t share it the way we did in the decades after the war.” 


For an array of reports, analysis, and data on PIAAC, go to http://piaacgatewav.com . 
http://nces.ed.gov/survevs/piaac . and www.oecd.org/site/piaac . 
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2. The Good of PIAAC 


PIAAC provides a much better understanding about “the distribution of human capital and of 
proficiencies at the national and international levels.” And it’s a good complement to the many 
school-based surveys that are done. In the U.S., by the time students reach 12‘*' grade many have 
already left school. So this product of the schools is what we’re often looking at instead of the 
larger issue of human capital of the population overall. Unlike the school surveys, PIAAC is 
administered through face-to-face interviewing via computer in the homes of a nationally 
representative sample of adults. And for the first time in a large-scale international assessment, 
reading components have been incorporated. 

“PIAAC adds another dimension to our understanding of how well we’re doing and how well 
our schools are doing.” The data sheds light on the extent to which skills translate into better 
opportunities for individuals and economies. “It can help us evaluate how effective our 
education and training system is and also how well our social workplace practices develop the 
required skills and proficiencies,” said a member of the PIAAC team. 


3. Changing Trends in Routine and Non-Routine Job Tasks: What It Means 

Between I960 and 2009, according to research by D.H. Autor and B.M. Price (2013),'^ we’ve 
seen “a significant decrease in the amount of tasks in jobs that require routine skills, manual or 
otherwise.” Since 1980 three on-the-job occupational routines have steadily declined: manual 
tasks of a routine and non-routine nature, and routine cognitive tasks. During this same period, 
heavy increases have occurred for non-routine analytic and non-routine interpersonal tasks, or in 
everyday language, in problem-solving, critical thinking, and communications skills. 

In effect, jobs have become more complex, the structure of the economy has changed, and so 
have the outcomes. This makes research on trends and gaps in lifetime earnings all the more 
important. Between 1978 and 2008, according to the PIAAC team, the lifetime earnings of male 
high school dropouts dropped by about 34%. We haven’t made much progress. If you’re a male 
high school dropout today, “you still see your expected lifetime earnings drop by 28%, whereas 
with an associate degree it’s 16% and with a bachelors there’s again of about 4%. The people 
who have benefited the most are those with master’s degrees. They have had a 33% increase. 
What we’re seeing is “this hollowing out of the middle class” — a very real outcome! 

People with higher levels of education are increasingly advantaged in U.S. society — ^not 
“because they’re making so much more, but because others are making so much less.” This in 
turn is a function of what employers are willing and able to pay in a global economy where they 
have a larger pool from which to hire. 


Autor, D.H. and B.M. Price (2013), “The Changing Task Composition of the US Labor Market: An Update of 
Autor, Levy, and Murnane (2003)," MIT Mimeography, June 2013. 
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4. Mediocrity & Increased Inequality 

PIAAC was administered to some 166,000 adults in 24 participating countries ranging from 16- 
65 years of age. These adults represent some 724 million people living in households in these 
countries. Here are a few of the specific U.S. findings'^ as highlighted at the Roundtable: 

• The U.S. performed significantly below the international average in Literacy, Numeracy, 
and a new ground-breaking scale called Problem Solving in Technology Rich 
Environments. About one in six adults in the U.S. scores at level 3 and lower on the 
Literacy measure. One of every three adults is at level 3 and below in Numeracy. 

• Older adults are doing better than the international average on the Literacy measure, but 
16- to 24-year-olds, who will be in the job market for the next 30 years or more, score 
much lower than the international average. 

• The U.S. ranked last in terms of the percentage of 16-24 year olds who performed in 
levels 2 and 3 (the two highest levels) on the Problem Solving measure. 

• Blacks and Hispanics are 3-4 times more likely to have poor skills than whites. 

• In the U.S., a greater percentage of those who were bom outside of the country perform 
at the lowest proficiency levels in Literacy and Numeracy than those bom here. 

• U.S. skilled workers had the same percentage of top performers in Literacy as the 
international average, but unskilled workers had a greater percentage of low performers 
than internationally. 

• The U.S. average literacy score in PIAAC (2012) is not significantly different than in 
2003, but it is lower than in 1994. Our 2012 average numeracy score is lower than 
in 2003. 

It is clear from just this short list that the U.S. findings stand as a huge challenge for the U.S. 
economy in terms of wage stagnation and rising inequality. It also suggests that serious thought 
should be given by policy makers to refocusing investments more on the 16-24 age group 
because “older generations are moving out of the job market, and younger generations are 
coming into it.” 

It was observed during the PIAAC discussion that some countries are doing a much better job 
of adapting to changes in the world labor force than others. “The U.S. is living off its past 
investments and not doing enough either currently or in thinking about future investments.” 


See Appendix IV. 
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Finally, the concluding set of Roundtable comments on the challenge of PIAAC for the U.S. is 
worth including verbatim here: 

“If Fm a policymaker and looking into the future I have to come to two conclusions. One is that 
our schools are failing large segments of our population. The other is that if we care about 
income distribution and distribution of skills we have to be worried about the PIAAC findings. 
Our best, the top 20%, are as competitive as any country in the world. But it’s the gap between 
the best and the worst that drives down the average and creates the inequality. 

“If we believe that the U.S. economy can go forward on the top 20% and that they can drive the 
economy, which some people seem to believe, we can ignore the PIAAC findings. But if we 
care about the other 80% [30 million or more adults over age 16 tested at the lowest levels], if we 
care about not wanting them to be behind, if we care about the fact that our kids are growing up 
increasingly in poverty, then it is clear that major investments are needed to reverse inequality 
and the education declines we are showing. We ought to care about ‘the distribution.’ We 
should want to bring up the bottom!” 

G. ROI IN CANADA’S ADULT EDUCATION 

In Canada, ROI for adult education investment is part of the National Economic Action Plan. 
Although the country doesn’t have a department of education as such, its forward-thinking 
minister of employment (equivalent to our secretary of labor) understands the adult skills 
challenge there. He provided funding to the Conference Board for a new Center on Skills and 

1 O 

Postsecondary Education. Our Roundtable participant holds high-level responsibilities at the 
Center, which is a five-year independent, nonpartisan initiative to examine and plan for Canada’s 
advanced skills and education needs. 

The Center does not advocate, but influences the national dialogue through “evidence” and hard 
facts. “We aim to get the attention of a broad canvas of people, something the U.S. also should 
do.” The results of PIAAC make it clear that “we are failing ourselves importantly,” where the 
skills and capacity of our people are concerned. The Center is a major new approach for 
addressing this challenge. 

I. ROI is Needed to Get Buv-In 

Robert Reich said “the only unique asset a business has is its workforce,” which should 
automatically be a basis for investing in human capital development. But several problems make 
this difficult for Adult Education, according to our panelist. For one thing, he said, employers 
don’t trust human capital development as much as machines or equipment because of high 
employee turnover. “Machinery and equipment cannot resign and move elsewhere,” some of 
them say. In addition, “the average Canadian in the mid-1970’s was 28, today the average 
Canadian is near 40. In many countries, including Spain, Italy, Russia, and Japan, people are 
getting older too. But in others, like India, the working population is young. Another force at 
work in the U.S. and Canada is that so many people cannot afford to retire, so “we’re going 


For information about the Center, go to http://www.conferenceboard.ca/spse/default.aspx . 
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backwards in retirement age.” A fourth factor is that employers and government officials often 
make decisions on an emotional basis rather than through careful analysis of actual experience, 
such as the cost impact to business from underutilization of workers. 

ROI is considered by the Center and its Conference Board parent to be “part of a suite of tools” 
that is needed to put this kind of intelligence together with understanding of the emotional 
responses that the various players have. “We don’t just show GDP changes when we make the 
case with government and business, but we also provide evidence in terms of lost tax revenue 
due to underutilization, and other cost factors.” Various studies in Canada have shown huge 
economic costs attributable to underutilization of adult skills. 

The important point is that in Canada the case is made strategically and with understanding that 
buy-in is needed from all sorts of people, individuals as well as business and government. 

2. Everybody Cares About Something 

Employers in Canada actually do care about many outcomes, our panelist said. They want 
evidence of many things — like how to deconstruct work and find out if it changes time-on-task, 
or error rates, or cycle times. They need to know about and keep data on on-the-job accidents 
and absenteeism, and want to know about such soft skills as worker satisfaction, employee 
confidence, and empowerment. They want to know about the depth of employees’ computer 
skills, and about how “communications courses” for employees (which may involve literacy 
instruction) impact on safety and productivity. If Adult Education can access or develop 
information on these and other things employers care about, if they can factor in employee or 
workforce cultural behaviors, and if they can then put the pieces together in a persuasive relevant 
way, this will do a lot to achieve buy-in. 

In Canada, health costs are going through the roof, as they are in the U.S. Changing health 
outcomes for people through education and skills upgrading is “a lot more efficient and effective 
than delivering more health care.” This is an area of high interest in the country, and literacy is 
being linked to health and safety in ways that influence the government agenda. Healthier 
people obviously need less care — ^research shows a strong connection between education level 
and health status. 

Still another area of high interest is to tap the role of unions, which have a long history of 
successful programming. “Unions actually care about this and can be significant players in some 
sections of the economy.” 

3. Changing Job Expectations 

Studies show that people around the world who used to put up with drudgery and repetitiveness 
in their jobs are not willing to do that anymore. In the U.S., research indicates that only 25% of 
our workers are challenged on their job, and 50% of American workers say they don’t like their 
jobs, an all time high! When asked what they plan to do about it, 60% of those workers said they 
plan to quit. 
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One reason suggested for this phenomenon is that the social media and various other 
technologies have given many people greater access to skills and education development and 
this is not being caught by planners. It was observed that every child texts today, so it isn’t 
necessarily that they can’t read and write, maybe they just don’t want to. We haven’t examined 
that. People are more accomplished in some ways that they used to be and thus have a higher 
expectation of jobs, which, even for college graduates, is not being met. They also know more 
about the world from the outreach of our communications technology and they know what kind 
of jobs are available in other countries. In short, it’s not a zero sum world any longer. Special 
planning needs to be done around this issue! We haven’t even begun to think about this 
new challenge. 

H. RECOMMENDATIONS 

CAAL’s STEPPING UP report ended with several recommendations to advance ROI 
understanding and practice. They are directed to federal and state government; research, policy, 
and advocacy; building business involvement and awareness; getting ready to document WIA 
ROI measures at the state level, and correcting NRS data limitations that impede state-level ROI 
work. To supplement that first set of ROI recommendations, the Roundtable concluded with 
several more ideas: 

I. Use PIAAC 

PIAAC gives us extremely valuable data to guide advancements in teaching and learning, and 
to better define quality and effectiveness for our core constituencies. We need to draw on the 
PIAAC data and analyses as we shape the common taxonomy necessary for messaging and 
communicating ROI in a way that reflects program purposes, beneficiaries, and outcomes for 
various groups. 

2. Develop State Databases 

We are making progress in longitudinal database development across the states, but we need to 
give closer attention to developing coherent, integrated, and accessible databases that generate 
consistent, comprehensive data analyses at both the state and national levels. 

We should be able to generate data on the variety of measures that push the state data systems — 
in the areas of adult education, postsecondary, and workforce programs. A key goal should be to 
track students from one program type to another and to enable significant wage record matches. 

Almost all states receive federal resources for developing their longitudinal data systems (EDS). 
The focus on incorporating Adult Education into that system is not as strong as it should be. 
Participants recommended that every state pay attention to this oversight due to the critically 
important role of Adult Education in moving all categories of low-skilled adults toward readiness 
for college and employability. 
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3. Get the Message and Definitions Right 


Throughout the discussion, the importance of effective and consistent messaging was stressed. 
To this end we need to draw on PIAAC, as noted above, and to engage in activities that move us 
toward common agreement on what the important ROI indicators are for different service 
contexts and program goals. We need to make it everyone’s business to be sure that we all know 
from the start what we will get out of funding and participating in specific programs. Business 
and community college groups need to be at the table in such planning and evaluation activities. 

Across the board in Adult Education there should be special focus on determining the key 
measures that relate to ROI for serving special-needs groups — ^not just those represented at 
the Roundtable, but those who plan to enter the workforce, homemakers, incumbent workers, 
and others. ROI database development should reflect all of the groups. 

We also must better define and rethink the skills that employers need and say they need in order 
to devise appropriate ways to measure those skills. One participant suggested that state and local 
planners redirect some of the effort they now focus on legislators to employers. Employers 
know what the pipeline looks like. They know what credentials already exist in the market. 
Employer-based data would allow programs to help clients make better decisions about the 
education and training they pursue again at all ages. For example, it is important to determine a 
program’s value to the employer, such as added productivity and surge costs. But most ROI 
calculations do not now do this very well. Nor do they factor in cultural behavior skills that have 
an impact on job performance. 

4. Develop and Support the Research Agenda 

We need to develop cost modeling tools in the context of ROI that focus on macro-economic 
impacts related to govemment(s), sectors, and private sectors and firms. PIAAC suggests many 
new areas for research, including secondary analysis of the PIAAC data itself. 

We should involve a wide variety of players in the research — the research universities (to begin 
to develop our next generation of researchers'^), the National Governors Association, national 
and state chambers of commerce, the National Association of Manufacturers, workforce 
investment boards, and other groups that understand data analytics. 

WIA Title II funding levels and program goals provide an essential framework for what the 
states do in adult education and workforce skills development, but state funding for those 
education activities is many times greater than federal dollars. So both federal departments and 
the states must be able to show Congress and their own legislatures how their investments in 


The American Institutes of Research was present at CAAL’s ROI Roundtable. A recent AIR initiative picks up 
on this Roundtable recommendation through a new research program on topics related to PIAAC. Successful 
competitors for $8,000 grants will be presented at an invitational PIAAC research conference in late 2014. 
Proposals are due March 28, applicants will be notified of awards in mid-April, and the papers will be due by the 
end of September. For more information, go to https://communitv.lincs.ed.gov/discussion/researching-and- 
reporting-piacc-data . 
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adult education and higher level education/training result in measurable social and fiscal gains. 
Research that enables them to develop that capacity is important— see Appendix III for two recent 
CAAL papers (items 1 and 2 on page 30) that illustrate the power of such data. Among other 
variables, states should be able to do evaluations that calculate the net gain of participation in 
programs to the students and the net value to the taxpayer. 

The fact that we have done little to understand the serious challenge of social media, especially 
its use by young people, affects planning for Adult Education. We need research on this topic. 

Finally, one participant commented that a question we never get to in our research is whether 
it’s the program or the policy that produces a desired outcome. He urged impact studies that 
shed light on this matter. 

5. Involve Employers, Business Associations, & Unions 

CAAL and many other research and policy groups have stressed in recent years how critical it is 
for adult education programs to partner with employers, business associations, workforce 
development agencies, and union programs. This was another recurring theme of the Roundtable 
and it is evident in many of the suggestions given above. But beyond these, the participants made 
two specific suggestions: 

• Undertake activities that help manufacturing associations, workforce boards, union programs, 
and other business related entities better understand ROI data analytics. Then link this 
research and retum-on-investment to innovative strategies to deliver services. 

• Give priority to identifying common measures between agencies and organizations that are 
involved in the same kind of workforce and literacy efforts. 

6. Involve Governors 


Many ROI leadership states have sought and received valuable support from their governors. 

A governor’s support for workforce education and ROI development is critical to the success of 
programs, contributions to state agencies, and political recognition of program outcomes. 

7. Face Funding Realities 

Much of the vital work outlined above will not be possible without new funding, a reality we 
need to face squarely. Substantial new funding will be required from government, philanthropy, 
and business to develop and spread exemplary program services and good ROI models and to 
support the research we should undertake to advance service and related ROI. Some of the top 
priorities are: 

• Good service and ROI models need to be identified and then scaled up. CAAL’s 2013 ROI 
survey and the Roundtable discussion identified examples of both. But there are many other 
candidates worthy of consideration. Some of the participants called for activities that are 
innovative in the way services and ROI activities are carried out, and many recognized the 
importance of collaboration. 
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• “Dual generation” programming was a theme touehed on throughout the discussion as 
needing higher priority. Several participants stressed that we should fund projects related to 
parents and children together in dual generation settings, and for enhancing the employability 
of parents. Some noted that this will especially important if/when a new immigration bill 

is enacted. 

• The growth of credentialing at the state and national levels was also frequently mentioned. 
We need accelerated programs for people with higher levels of education, even when their 
English is minimal. We should be trying to help them get credentialed and then make sure 
they are ending up in good jobs. But more generally, we need to invest in interventions that 
combine adult literacy, career pathways for lower-level literacy adults, bridge programs, and 
integrated education in which stackable credentials are an important feature. 

• Much of the development and programming work discussed in this paper must go hand-in- 
hand with professional development of teachers and leaders. Special projects should be 
funded to address and reward professional development needs. 

• In addition to national and state funding from traditional adult education sources, as Adult 
education focuses more and more on moving adults to readiness for college and jobs — which 
requires coordination with other kinds of service organizations — states should be giving 
more attention to defining nontraditional funding streams. This includes those that target 
out-of-school youth 16-24 years of age, corrections education, unions, workforce 
development generally, and community colleges. 

In pursuing nontraditional funding approaches, one participant pointed out that the relationship 
many politicians have to ROI is inconsistent. They ask for ROI evidence all the time but they 
often ignore it. It is a special challenge for program planners and providers that policymakers 
often do not understand how hard it is to actually collect ROI data, he said. Sometimes they 
impose unrealistic deadlines and rarely do they want to pay for it. “In my experience, people 
can either run a program or collect data, but they are usually hard pressed to do both at the same 
time.” So while ROI is increasingly a precondition for funding, it doesn’t guarantee it. This 
is another reason to be thinking about ways to fund Adult Education programs beyond the 
traditional federal or state appropriations routes. 

I. CONCLUSION 

By its very nature, any full day of discussion about Return on Investment (ROI) in Adult 
Education is apt to be complex, even sometimes confusing. Often in the CAAL Roundtable we 
found ourselves immersed in the entire adult education enterprise. Each issue we probed shed 
new light on our topic but it also opened the door to many others that we could not fully address 
due to time constraints. But several messages were loud and clear: 

• Adult Education is undergoing fundamental restructuring and needs to do even more of it. 

As we go forward in the Adult Education enterprise in America, we need to think differently 
about the focus and outcome of our services and about the increasingly important role of ROI 
evidence. 
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• A good deal of adult education and training is going on in this country that is neither widely 
enough recognized nor documented, and for which the cost and overall ROI are not known. 
So we have much to learn. We need to understand ROI in Adult Education far better than 
we do. 

• It is important to raise the profile of Adult Education in national and state thinking. Many 
of the measures proposed as a result of the discussion will help achieve that. 

• In its own right as a system, Adult Education needs its own ammunition, which should be 
just as powerful as K-12 and higher education. PIAAC and ROI data are tools to that end. 

The purpose of this entire CAAL ROI effort was and is to extend and advance adult education 
and workforce skills services to millions of adults, way beyond the small number now being 
served. With the help of many experienced professionals, this paper has suggested some next- 
step actions to help move us in that direction. But it would be folly to think that significant 
service outreach and related ROI work can be achieved without substantial new funding. 

The data available through PIAAC and the research and policy work carried out for years by 
many national leadership groups including CAAL make a powerful case. In his concluding 
remarks, one researcher stated: “We’ve got to get our children off to a good start, and it’s one of 
the best investments we can make. But we’ve got a problem with our adults now, we’ve got so 
many people in need. Let’s get our act together. We need substantial new investments in Adult 
Education.” 

It is obvious that we have a huge skills gap in America, and that it fuels poverty and inequality. 
So, whether we’re talking about immigrants, or low-skilled adults of any age, or Blacks and 
Hispanics, or low-income people, or any other group in need of services, we need to pay 
attention and step up to the challenge. ROI evidence can help us do that. 

And, finally, the best should not be the enemy of the good! Our ROI data may not be perfect at 
this stage but much of it is very solid and useful. What we have is here to be used, and we can 
and need to act without delay! ■ 
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APPENDIX III 


RESOURCES 


1 . Labor Market Experiences, Earnings, Income Inadequacy Problems, and Civil Behavior of 
U.S. Adults by Educational Attainment: Consequences for Adult Education Programs 

This 22-page paper, prepared for CAAL in February 2014 by the Center for Labor Market 
Studies of Northeastern University (Ishwar Khatiwada, Andrew Sum, and Sheila Palma), is an 
update of information provided in 2008 in Reach Higher, America, the final report of the 
National Commission on Adult Literacy. http://www.caalusa.org/EdLevelsSocialOutcomes.pdf 


2 . Net Annual Eiscal Contributions of U.S. Adults Aged 18-64 by Education Attainment, 
2009-2012 

This paper, prepared in January 2014 for CAAL by the Center for Labor Market Studies of 
Northeastern University, provides analysis of the fiscal gains to the nation and to individuals of 
high school and college graduation in America for adults aged 18-64. It also presents aggregated 
and state-by- state data for 15 states: CA, CO, FL, IL, KY, MD, Ml, MN, NJ, NY, OH, OK, PA, 
TX, and VA. The paper is a partial update of data presented in Reach Higher, America. 
http://www.caalusa.org/NetFiscalContributionsQ9-12.pdf 


3 . In This Together: The Hidden Cost of Young Adult Unemployment 
Young Invincibles, January 2014 

This new report determines that, on average, one unemployed 18- to 24-year-old will cost the 
federal and respective state government over $4,100 annually in forgone tax revenue and benefits 
received. The costs to government grow as unemployed individuals age. On average, they 
estimate each unemployed 25- to 34-year-old will cost his or her federal and state government a 
staggering $9,875 annually. http://vounginvincibles.org/wp-content/uploads/2Q14/Ql/In-This- 
Together-The-Hidden-Cost-of-Young-Adult-Unemplovment.pdf 


4. Making a Market for Competency-Based Credentials 
Corporation for a Skilled Workforce 

The CSW has embarked on a multi-year initiative to increase the quality and use of competency- 
based credentials across the country. They are collaborating with leaders committed to realizing 
the potentially game-changing result of large-scale use of competency-based credentials by 
businesses, educators, workers, and students across the nation. Their new report is aimed at 
articulating a foundation for our collective work by analyzing the credentialing landscape, 
identifying promising credentialing approaches, and assessing what is needed to create a clear 
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and compelling credentials marketplace. Making a Market for Competency-Based Credentials 
compiles what they and the many leaders who informed this narrative have learned to date. 
http://www.skilledwork.org/sites/default/fdes/MakingaMarketforCompetenev- 
BasedCredentials.pdf 


5 . Developing Skilled Workers 

The Manufaeturing Institute of the National Institute of Manufaeturing 

This is a reeently-released toolkit for manufaeturers on reeruiting and training a quality 
workforce. It includes steps to take in partnering with business associations, community or 
technical colleges, and local workforce investment boards. 
http://browndigital.bpe.com/publication/?i=l 87979 


6 . The Potential and Value of Using Digital Badges for Adult Learners 
American Institutes for Researeh 

At their most basie level, digital badges are a new way to eapture and communieate what an 
individual knows and can demonstrate. This report examines the nature, value, and potential 
impaet of digital badges, an emerging electronie form of recognition of an individual’s 
knowledge and skills. Badges ean represent different levels of work and engagement, ineluding 
more granular skills or aehievements, marking in some cases small and/or very specifie abilities. 
For this reason badges hold particular promise for eertifying the skills of adult learners in basie 
education programs, many of whom have few, if any, formal eredentials (such as diplomas), but 
who are obtaining functional skills that would be valued in a workplace setting if a meehanism 
for eertifying those skills and knowledge was available. 
http://lincs.ed.gov/publieations/pdf/AIR Digital Badge Report 508.pdf 


7 . Measuring Alternative Educational Credentials: 2012 

The U.S. Census Bureau reports that in fall 2012, more than 50 million U.S. adults, or one in 
four, had obtained a professional certifieation, lieense or edueational certifieate apart from a 
postsecondary degree awarded by eolleges and universities. This is the Census Bureau’s first- 
ever report on this topie. Among the adults ineluded in the report, 12 million had both a 
professional eertification or lieense and an educational certificate; 34 million had a professional 
certification or license; and 7 million had an educational certificate. Getting an academie degree 
is not the only way for people to develop skills that pay off in the labor market. 
http://www.census.gov/prod/2Q14pubs/p7Q- 
138.pdf?eml=gd&utm medium=email&utm source=govdelivery 
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8 . Evaluating the Effectiveness of Correctional Education - A Meta-Analysis of Programs 
That Provide Education to Incarcerated Adults 
RAND Corporation 

RAND examined the current state of correctional education for incarcerated adults and juveniles 
and where the field is headed, which correctional education programs are effective, and how 
these programs can be implemented across different settings. 

http://www.rand.org/content/dam/rand/pubs/research reports/RR200/RR266/RAND RR266.pdf 


9 . Making Workforce Data Work: How Improved Education and Workforce Data Systems 
Could Help the U.S. Compete in the list Century Economy 
Workforce Data Quality Campaign (WDQC) 

This signature report lays out the WDQC’s policy priorities and includes real-world examples of 
the ways data is helping to strengthen education, governance, and business. It is designed to 
make a compelling case for improving education and workforce data, and to give state and 
federal leaders actionable ideas to move forward, http://www.workforcedqc.org/resources- 
events/resources/making-workforce-data-work 


10 . Literacy, Numeracy, and Problem Solving in Technology-Rich Environments Among 
U.S. Adults: Results from the Program for the International Assessment of Adult 
Competencies 2012 

This report presents results of the Program for the International Assessment of Adult 
Competencies (PIAAC). Results are reported for a representative sample of adults in the United 
States age 16 to 65 and are compared to an international average of adults in countries/regions 
that participated in the PIAAC 2012 assessment. The report presents average score results for 
three separate scales: literacy, numeracy, and problem solving in technology-rich environments 
and percentages of adults performing at different proficiency levels for each scale. Literacy and 
numeracy results are reported at Below Level 1, Level 1, Level 2, Level 3, and Level 4/5; 
problem solving in technology-rich environments is reported at Below Level 1, Level 1, Level 2, 
and Level 3. The report includes results for groups of adults as defined by various demographic 
characteristics (e.g., gender, age, race/ethnicity, and level of educational attainment) and level of 
skill use in and outside of work. Overall results in literacy and numeracy are compared to results 
from previous international assessments. http://nces.ed.gOv/pubs2014/2014008.pdf 

11 . Time for the U.S. to Reskill? What the Survey of Adult Skills Says 

The basic skills of literacy and numeracy are of huge importance to our economies and societies. 
The OECD’s new Survey of Adult Skills (PIAAC) assesses skills of literacy, numeracy and a 
newly assessed domain of “problem solving in technology-rich environments” in a number of 
countries. This special report, published alongside the main international survey, looks at the 
results for the United States and identifies their policy implications. 
http://skills.oecd.org/Survev of Adult Skills US.pdf 
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12. Performance of U.S. 15-Year-Old Students in Mathematics, Science, and Reading 
Literacy in an International Context-First Look at PISA 2012 

First Look at PISA 2012 reports average seale scores and the percentage of 15-year-old students 
reaching selected proficiency levels, comparing the United States with other participating 
education systems. Results for three U.S. states are also reported. 
http : //nces . ed. gov/pubsearch/pubsinfo. asp?pubid=20 14024 


13. Help Wanted: Projections of Jobs and Education Requirements Through 2018 
Center on Education and the Workforce, Georgetown University 

This report presents a new approach to answering some critical questions about the emerging 
economy, including: 

• When will the jobs come back? 

• Where will the jobs be? Which states? Which industries? Which occupations? 

• What postsecondary certificates and degrees will be required? 

• Will the education system be able to keep up? 

• How much will it cost to fund the postsecondary education America needs? 
http://cew.georgetown.edu/iobs20 1 8 


14. Using Return on Investment (ROI) and Other Related Tools: Guidelines for Measuring 
Career and Technical Education (CTE) Internal Efficiency and External Effectiveness 
National Research Center for Career and Technical Education 

This 70-page publication, funded by the Office of Career, Technical, and Adult Education 
(OCTAE) of the U.S. Department of Education, addresses many of the points discussed in 
CAAL’s November 8* ROI Roundtable. It was published in June 2012 and written by Pradeep 
Kotamraju and John L. Mettille. 

http://www.nrccte.org/sites/default/files/publication-files/nrccte roi guide 
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APPENDIX IV 


Materials From CAAL’s 
November 8, 2013 ROI Roundtable 


Adult Education in Arkansas 

Including the Workforce Alliance for Growth in the Eeonomy (WAGE) 

OECD Education and Online Skill Assessment 

(Information about how it works and who can benefit from using it) 

PIAAC Slide-Set Presentation 

(the Program for International Assessment of Adult Competencies) 
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Workforce Alliance for Growth in the Economy™ 
Competencies 


C1 

C2 

C3 

C4 

C5 

C6 

C7 

C8 

C9 

C10 

C11 

C12 

C13 

C14 

C15 


Communication 


Speaking clearly and using language easily understood by the listener. 

Speaking to inform another or to provide information. 

Speaking to inquire or to collect information. 

Speaking to persuade another. 

Speaking politely, with respect to cultural diversity, regardless of 
personal feelings. 

Speaking at a rate easily understood by the listener. 

Speaking at a volume appropriate of the circumstances and the 
message. 

Structuring spoken inquiries so that listener responds appropriately. 
Restructuring a message, when necessary, to facilitate listener 
understanding. 

Using proper telephone etiquette. 

Using task-related words in the proper context. 

Using tonal expression to facilitate communication, cooperation, and 
goodwill. 

Using vocabulary appropriate to the circumstances and the message. 
Understanding the meaning of and possessing self-awareness of body 
gestures. 

Understanding the meaning of and possessing self-awareness of facial 
gestures. 


C16 Understanding the meaning of and possessing self-awareness of 
posture. 

C1 7 Understanding the meaning of and possessing self-awareness of 
tonal expression. 

C1 8 Listening to advise, assist, or enable. 

C1 9 Listening to facilitate cooperation, goodwill, or teamwork. 

C20 Listening to learn or understand. 

C21 Listening for nonverbal clues, tonal expression, emotions. 

C22 Listening to obtain information to solve problems, make decisions. 

C23 Listening for omissions of information. 

C24 Recognizing the main intent of a spoken message. 

C25 Recognizing and organizing details of a spoken message. 

C26 Recognizing probable meaning of unclear communication by using 
context clues. 

C27 Screening irrelevant information and distractions. 

C28 Sorting relevant from irrelevant information. 

C29 Verifying accuracy by restating/repeating message. 

C30 Verifying comprehension by asking questions. 


Math 

M1 Reading, writing, and counting singie and muitipie digit whoie 
numbers. 


M2 Adding, subtracting, muitipiying, and dividing singie and muitipie 
digit numbers. 

M3 Using addition, subtraction, muitipiication, and division to soive 
probiems. 

M4 Rounding off singie and muitipie digit numbers to compiete a task 
or subtask. 

MS Comparing whoie numbers to determine variations. 

M6 Reading and writing common fractions to compiete a task or 
subtask. 

M7 Adding, subtracting, muitipiying, and dividing common fractions 
to soive probiems. 

M8 Comparing decimai and proper fractions to determine variation. 

M9 Carrying our arithmetic computations invoiving doiiars and cents. 

M10 Reading and writing decimais to one or more pieces to compiete 
a task or subtask. 

M11 Rounding off decimais to one or more pieces to compiete a task. 

M12 Adding, subtracting, muitipiying, and dividing decimais to one or 
more piaces. 

M1 3 Reading, writing, and computing percents to complete a task or 
subtask. 

M14 Using percents to determine increase or decrease. 


M15 Converting fractions to decimais, percents to fractions, 
fractions to percents, percents to decimais, common 
fractions or mixed numbers to decimai fractions, and 
decimais fractions to common fractions or mixed numbers 
to compiete a task or subtask. 

M16 Soiving probiems by seiecting and using correct order of 
operations. 

M17 Computing averages, ranges, means, medians, rations, or 
proportions. 

M18 Reading numbers or symbois from time, weight, distance, 
and voiume measuring scaies. 

M19 Using a measuring device to determine an object’s weight, 
distance, and voiume in standard or metric units. 

M20 Performing basic metric conversions invoiving weight, 
distance, and voiume. 

M21 Using a caicuiator to perform basic arithmetic operations to 
soive probiems. 

M22 Determining if a soiution to a mathematicai probiem is 
reasonabie. 

M23 Estimating answers to a mathematicai probiem when an 
exact answer is not needed. 

M24 Reading a bar, iine, circie, or other graph to anaiyze interpret 
or compare data points. 

M25 Using descriptive statistics to describe data. 

M26 Soiving probiems using a systematic method. 

M27 Applying geometric functions to determine the properties, measurement, 
and relationships of points, lines, angles surfaces, and/or solids. 


WAGE™ is an Arkansas Department of Workforce Education, Adult Education Division program. 


Reading 

R1 Recognizing and using task-reiated words, abbreviations, acronyms, R17 
and codes. 

R2 Identify factual details and specifications within a text. R18 


R3 Following sequential procedural directions to complete a task. 

R4 Determining the main idea of printed media or an activity. 

R5 Using table of contents, indices, or screen menus to iocate information. 

R6 Locating pages, titles, paragraphs or charts needed to answer 
questions or solve problems. 

R7 Skimming or scanning to determine whether text contains relevant 
information. 

R8 Cross-referencing within a document or program screens to locate 
information. 

R9 Using a completed form to locate information to compete a task. 


R19 

R20 

R21 

R22 

R23 

R24 

R25 


R10 Combining task-reiated information from muitiple sources. 

R1 1 Selecting parts of text, visual materials, or an activity to complete 
a task. 

R12 Identifying similarities and differences in objects. 


R26 

R27 

R28 


R13 Determining the presence of a defect or damage. 


R29 


R14 

R15 

R16 


Classifying or matching objects by color, size, or significant 
marking. 

Classifying, sorting, or arranging documents. 

Distinguishing between relevant and irrelevant information in text 
or visuals. 


R32 

R33 

R34 


W1 Spelling task-related words and abbreviations correctly. 


Writing 

I W11 


W2 Writing key technicai words and abbreviations correctiy. 
W3 Writing symbols accurately. 

W4 Keyboarding accurately. 


W12 

W13 

W14 


W5 Entering appropriate information onto a form. 


W15 


W6 Recording essential information that involves more than one sentence. 


W16 


W7 Recording essential information in phrases or simple sentences 
accurately and precisely 

W8 Transferring numbers, codes, dates, and figures from written sources 
onto appropriate sections or a form. 

W9 Writing a report including necessary support documentation or 
classification. 

W10 Writing brief, descriptive accounts of activities or transactions 
performed. 


W17 

W18 

W19 

W20 

W21 


Using common knowledge for safety. 

Applying preventative measures prior to task to minimize 
problems. 

Knowing appropriate procedure for emergencies. 

Selecting appropriate course of action in emergency. 
Reading two or more column charts to obtain information. 
Locating chart information at intersections of rows and 
columns. 

Cross-referencing charted information within printed media. 

Applying information from tables, graphs or flow charts to locate 
malfunctions or selected actions. 

Using flow charts to sequence events, arrive at a decision, or 
problem solve. 

Identifying components within a schematic. 

Isolating problem components in schematics, tracing to cause of 
problem. 

Identifying details, labels, numbers, parts of an illustration, 
parts from a key or legend. 

Following sequenced illustrations as a guide. 

Making inferences or drawing conclusions from printed media. 

Interpreting codes and symbols. 

Identifying objectives, intent, and all essential and supporting 
details of a document. 

Outlining a situation by identifying key ideas and supporting 
details. 

Summarizing essential details for a written communication. 
Selecting relevant details for a written communication. 

Stating general impressions of an event or situations as they 
relate to specific reporting goals. 

Summarizing events and precise dialogue in an accurate, 
complete, and objective manner. 

Summarizing the major points presented in a written 
communication. 

Generating a written communication according to a specific 
format. 

Identifying objectives, intent, target audience of a written 
communication. 

Generating a written communication, arranging events 
sequentially. 

Generating written presentation of proposed courses of action, 
justifying one over the other. 

Appraising a written communication and making adjustments to 
improve clarity. 


Bolded items represent the WAGE™ program core competencies. 


WAGE™ is an Arkansas Department of Workforce Education, Adult Education Division program. 




Patricia Bates 

Chair, AACAE Legislative Committee 
870/314-8635 
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Why Adult Education is Needed 

• 491,863 Arkansans 18 years of age or older function below a 12**^ grade level 

• 268,781 working-age Arkansans have less than a high school diploma 

• 399,755 of Arkansans lack basic literacy skills 

• 79,002 Arkansans 5 years or older speak English “less than very well” 

• 12,500 students dropped out of high school in the 2008-2009 school year 

Primary Functions of the Adult Education Division 

• Provide basic academic education skills to pass the GED® tests 

• Conduct classes to improve basic academic skills for those who function 
below 12* *^ grade level 

• Provide instruction to improve basic academic skills to enable students to 
enter post-secondary education or training 

• Provide basic academic skills training to enhance employability 

> Career Readiness Certificate (CRC) 

> Workforce Alliance for Growth in the Economy (WAGE)^"’ 

• Provide English for Second Language (ESL) instruction 

Adult Education Results for 2008-2009 


• GED’s awarded 7,443 

• GED pass rate in U.S. is 73%; pass rate in Arkansas 85% 

• Students served 51 ,295 

• Students making grade level educational gains 79% 

• Students entering Post-Secondary 2,386 

• Students who gained Employment 4,804 

• Students who retained employment 3,362 

• WAGE Certificates Earned 707 

• Enrollment in ESL 7,189 

• State funds expended for 2008-2009 $18,025,693 

• Federal funds expended for 2008-2009 $ 5,269,613 


Economic Benefits to Arkansas Resulting from 2008-2009 Funding 

• Studies show that a high school graduate earns an average of $8,860 more per year 
than a non-HS graduate. The GED is a step that places the employee in this higher 
earnings category. Using 2008-2009 state investment of $18 million and applying 
these study results to the 7,443 GED’s gives 

> Projected Additional State Income Tax per Year (3% rate) $1 ,978,349 

> ($8,860 X 7,443 GEDs x 0.03) 

> Projected Additional State Sales Tax per Year (6% rate) $3,165,359 

> ($8,860 X 0.80 X 7,443 x 0.06) 

> Projected Total Annual State Tax Benefit from Added Earnings $5,143,708 

• Of the $18,025,693 expended for adult education, $14,400,000 (80%) is for salaries for 
adult education employees. The state realizes $1,123,200 directly back in income 
and sales taxes from these salaries each year. 

• According to the 2000 U.S. Census, individuals with some college earn an average of 
$4,290 per year more than high school graduates. In 2007-2008, 2,386 students from 
adult education entered post secondary education. This generates $10,235,940 in 
taxable revenue resulting in $921,234 annually in additional state taxes. 

• GEDs earned by prison inmates directly reduce recidivism by 6.1%. 850 GEDs were 
earned in Arkansas prisons in 2005. The lower recidivism rate means that 52 of 
these do not return to prison. This means an annual savings to the state of 
$1,138,800. Also, recent legislation allowing 90 days meritorious good time for 
earning the GED produces further savings in housing cost of $4,400,000 due to early 
release of this graduating class of 850. These are annual benefits. 

• Using these annual benefits and assuming a conservative 10 year working career for 
the GED recipient, the net present value of state adult education funding, after return 
of the $18,025,693 state funding, for the 2008-2009 GED class is $26,642,618, 
generating a rate of return on investment of 43%. That is, the State of Arkansas gets 
all funding dollars back plus an additional $27 million for money invested in the 
Adult Education program in 2008-2009. Similar results apply to other years. 

• In addition to the above specific benefits, lifetime total health savings for the high 
school graduate, compared to a non-graduate, is estimated to be $40,500. In 2007- 
2008, 5,216 students aged 16-24 earned their GED in Arkansas. The total savings in 
public health care over their lifetimes is $211,248,000. Also, the lifetime total welfare 
cost- savings for a high school graduate is estimated to be $3,000. Again using the 
5,216 students aged 16-24 who earned their GED in Arkansas in 2007-2008, this 
would produce a total state welfare cost-savings over their lifetimes of $15,648,000. 


Educational attainment of parents positively impacts the family literacy environment. 
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EDUCATION AND SKILLS ONLINE ASSESSMENT 
The Online Version of PIAAC (forthcoming in January 2014) 
A joint Initiative of the OECD and the European Union 


WHAT IS EDUCATION AND SKILLS (E&S) ONLINE? 

> E&S Online is an assessment tool designed to provide individual-level results that 
are linked to the OECD Survey of Adult Skills (PIAAC) measures of literacy, numeracy 
and problem solving in technology-rich environments. All results are comparable to 
the measures used in the Survey and can be benchmarked against the national and 
international results available for the participating countries. 

WHO CAN TAKE THE ASSESSMENT? 

> E&S Online has been developed and 
validated for a population ranging from 16 to 65 
years. It can be used for adults of any age who 
want to demonstrate their workforce-readiness 
skills and benchmark themselves with adults of 
similar background in their country or 
internationally. Institutions, organisations or 
local governments can use the online tool to 
assess the skills of a particular population with 
the goal of providing training or for research purposes. 

WHAT IS MEASURED? 

> E&S Online measures a set of cognitive and non-cognitive skills that individuals 
need for full participation in modern societies. These skills and knowledge include 
being able to understand and use printed and electronic texts, reason with numbers, 
and solve problems using information and communication technologies. Literacy and 
numeracy skills have been shown to be important foundation skills needed for the 
development of higher-order problem-solving skills. The assessment also measures 
reading-component skills to better understand the difficulties faced by those who 
demonstrate poor reading literacy skills. It also includes non-cognitive skills modules 
that allow individuals or organisations to obtain information on the use of skills at 
work and at home, and respondents' health and well-being. 

WHAT INFORMATION WILL BE PROVIDED? 

> E&S Online will provide individuals and/or organisations with an easy-to-read 
report after the assessment is completed. These reports will characterise the 
strengths and weaknesses in each cognitive area assessed. Scores will be reported in 
terms of proficiency levels that are related to the complexity of the tasks performed. 

> E&S Online will also provide summary information in each non-cognitive area 
selected and tested. In addition to obtaining descriptive and summary information, 
individuals and/or organisations will be able to benchmark their results against the 
results of the OECD Survey of Adult Skills, both from their own country and 
internationally. 





OECD Survey of Adult Skills 
(PIAAC) 


The Survey is an initiative of the 
OECD that assesses the 
proficiency of adults in key 
information-processing skills 
essential for participating in the 
information-rich economies and 
societies of the 21^* century. 
These are: literacy, numeracy, 
and problem solving in 
technology-rich environments. 

The Survey allows countries to 
analyse the level and distribution 
of skills among their adult 
populations as well as the extent 
of skills use in different contexts. 
Thirty-three countries have 
participated in this survey, which 
will allow for international 
benchmarking. First results will 
be published in October 2013. 


Participating Countries 


Australia, Austria, Belgium, 
Canada, Chile, the Czech 
Republic, Denmark, Estonia, 
Finland, France, Germany, 
Greece, *lndonesia, Ireland, 
Israel, Italy, ^Lithuania, Japan, 
Korea, the Netherlands, New 
Zealand, Norway, Poland, *the 
Russian Federation, ^Singapore, 
the Slovak Republic, Slovenia, 
Spain, Sweden, Turkey, the 
United Kingdom, and the United 
States. 


OECD Partner Countries 






HOW IS THE ASSESSMENT ADMINISTERED? 

> E&S Online is a fully computerised measure of cognitive and non-cognitive skills. It is available on the Internet, 
and incorporates flexibility and adaptability to provide reliable and valid measures of critical skills associated with 
work, home and the community. 

> E&S Online allows individuals to choose which skills they want to assess. As a baseline, individuals receive a 
core set of background questions followed by a set of tasks that will provide information about their literacy and 
numeracy skills. They can elect to receive information about their reading-component skills, problem-solving skills 
and selected non-cognitive skills. 

> E&S Online is available in several languages: English (Canada, Ireland, UK, US), French (Canada, France), 
Spanish (Spain, US), Czech, Italian and Japanese. 


The Main Elements of the Education & Skills Online Assessment 


Demographic 

Characteristics 


Background Questionnaire 


^ , . Social and Linguistic ^ , 

Education Background Employment Status 




Core Cognitive Modules 



Literacy 


Numeracy 


Optional Cognitive 
Modules 


• Reading Components 

• Problem Solving in 
Technology-Rich 
Environments 



Optional Non-Cognitive 
Modules 


• Behavioral Performance 
Competencies 
•Subjective Well-Being & 
Health 

•Career Interest and 
Intentionality 
•Work/Training History & 
SkillsTransfer 


Background information 

The background questionnaire contains questions on 
demographic characteristics, social and linguistic 
background, education, and employment status. It 
has been designed to provide information on such 
issues as the relationships between education and 
training and skills development, and also to 
benchmark the performance of test-takers against 
that of individuals with similar backgrounds and 
experiences. 


Non-cognitive modules 

The non-cognitive modules of E&S Online provide 
information about aspects of the personality, 
interests and work history of test-takers. This 
information may help individuals to think about their 
occupational choices. These modules assess various 
factors that can have a direct impact on training and 
success in the workplace: 

► Behavioural Performance Competencies 

► Subjective Well-Being & Health 

► Career Interest and Intentionality 

► Work/Training History & Skills Transfer 







Cognitive modules 

The direct-assessment component of the survey evaluates the skills of adults in two core domains - literacy and 
numeracy - and two optional domains - reading components and problem solving in technology-rich environments. 
These are considered to constitute ''key" competencies since they provide a foundation for the development of 
other, higher-order cognitive skills and are prerequisites for gaining access to and understanding specific domains of 
knowledge. In addition, these skills are necessary in a broad range of contexts, from education through work to 
everyday life. 

Summary of the cognitive domains in Education & Skills Online 



Literacy 

Numeracy 

Problem solving in technology-rich 
environments 

Definition 

Ability to understand, evaluate. 

Ability to access, use, interpret 

Ability to use digital technology. 


use and engage with written 

and communicate 

communication tools and networks 


texts to participate in society. 

mathematical information and 

to acquire and evaluate 


achieve one's goals, and 

ideas in order to engage in and 

information, communicate with 


develop one's knowledge and 
potential. 

manage the mathematical 
demands of a range of 
situations in adult life. 

others, and perform practical tasks. 

Content 

Texts are characterised by : 

Mathematical content. 

Technology: 


Medium: 

information and ideas: 

► Hardware devices 


► Print-based 

► Quantity and number 

► Software applications 


► Digital 

► Dimension and shape 

► Pattern, relationships. 

► Commands and functions 

► Representations (text. 


Format: 

► Continuous or prose texts 
(narration, argumentation 

change 

► Data and chance 

graphics, video) 
Nature of problems: 


or descriptions) 

Representations of 

► Intrinsic complexity (number 


► Non-continuous or 

mathematical content: 

of steps, alternatives required 


document texts 

► Objects and pictures 

for solution, complexity of 


(tables, lists, graphs) 

► Numbers and symbols 

computation and/or 


► Mixed texts 

► Diagrams, maps, graphs. 

transformation, number of 


► (combination of prose and 

tables 

constrains) 


document elements) 

► Texts 

► Explicitness of the problem 


► Multiple texts 

(juxtaposition or linking of 
independently generated 
elements) 

► Technology-based displays 

statement (largely unspecified 
or described in detail) 

Cognitive 

► Access and identify 

► Identify, locate or access 

► Setting goals and monitoring 

strategies 

► Integrate and interpret 

► Act upon and use (order. 

progress 


(relating parts of text to 

count, estimate, compute. 

► Planning 


one another) 

measure, model) 

► Acquiring and evaluating 


► Evaluate and reflect on 

► Interpret, evaluate and 
analyse 

► Communicate 

information 
► Using information 

Contexts 

► Personal 

► Everyday life 

► Personal 


► Work-related 

► Work-related 

► Work-related 


► Community 

► Education 

► Society & Community 

► Education 

► Community 





WHO ARE THE POTENTIAL USERS? 


> Researchers who would like to have access to tests that allow for benchmarking to deepen and complement the 
results of their assessment efforts at a national or regional level. 

> Organisations concerned with adult literacy and numeracy that seek literacy level benchmarks to compare with 
results of training. 

> Universities, vocational education and training centres that can use E&S Online as a diagnostic tool for incoming 
students and their need for literacy courses. 

> Government organisations interested in assessing the learning needs of unemployed adults, prison populations 
or economically disadvantaged adults. 

> Public or private companies that want to use the results to help them assess and recruit employees and identify 
training opportunities for their workforce. 

> Students or out-of-school youth transitioning to post-secondary education/training, who can use the results to 
assess their individual course needs. 

> Adults of any age who either wish to re-enter an education or training environment or want to demonstrate 
their workforce-readiness skills. 

> Any individuals who might want to compare their results with national and international results or to determine 
whether their skills have improved over time as the result of continuing education. 


DID YOU KNOW? Some adults who do 
poorly on literacy surveys have been shown to 
have strong basic reading and writing skills, but 
are unable to apply these skills to real-life 
situations. 



Page Web 

Contact 

www.oecd.org/site/piaac 

William Thorn (William.thorn@oecd.org) 

www.facebook.com/OECDSkillsSurveys 

Marta Encinas-Martin (Marta.encinas-martin@oecd.org) 
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Setting a Context 
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• America along with most countries around the 
world has undergone signif leant change over the 
last several decades. 

• This change has been driven in part by 
globalization and the rapid development and 
diffusion of Information A communication 
technologies. 

• One consequence of these changes has been 
the Increasing importance of education and skills 
to Individuals as well as to society. 
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Mean task input in percentiles of 1960 distribution 
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rs in routine ai 

United States, 1960 to 2009 


70.0 

65.0 

60.0 

55.0 

50.0 

45.0 

40.0 

35.0 

1960 1970 1980 1990 2000 2006 2009 



Routine manual 
Non-routine manual 
Routine cognitive 
Non-routine analytic 
Non-routine interpersonal 


Source: Autor, D. H. and B.M. Price (2013), "The Changing Task Composition of the US Labor Market: An Update 
of Autor, Levy, and Murnane (2003)", MIT Mimeograph, June. 
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Trends and Gaps in the Mean Lifetime Earnings* 
of Males Ages 16-74 by Levels of Education 


Educational Attainment 

1978 

2008 

Dollar Change 

% Change 






HS Dropout 

1,419,200 

937,600 

-481,600 

-34% 

HS Graduate 

2,120,900 

1,528,100 

-592,800 

-28% 

13-15 yrs / Associates Degree 

2,339,000 

1,960,500 

-378,500 

-16% 

Bachelor's Degree 

3,033,100 

3,158,800 

125,700 

4% 

Master's Degree or Higher 

3,392,144 

4,506,386 

1,114,242 

33% 


Source: Center for Labor Market Studies; 2013 
* In constant 2008 dollars 
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Setting a Context (cohi 


• The theory of evolution tells us that it is not the strongest 
or the most intelligent of the species that survives. It is the 
one that is most adaptable. 


• Our success in adapting to the current changes here in 
America and around the world will likely determine both 
the rate and shape of our economic growth in the future 
and how widely we share this growth. 


• I believe the PIAAC data contain a lot of valuable 
information for policy makers and other key stakeholders. 
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For example, PIAAC is able to 

• provide a better understanding of the distributions of 
key skills and proficiencies both at the national and 
international levels? 

• shed light on the extent skills translate into better 
opportunities for individuals & economies 

• help evaluate how effective our education and 
training systems, and our social and workplace 
practices are in developing the required skills and 
proficiencies? 
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Better Skills 
Better Jobs 
Better Lives 

A STRATEGIC APPROACH TO SKILLS POLICIES 



®))OECD 


skills.oecd; building the right skills 
and turning them into batter jobs and better lives 
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Key Facts about the Survey 
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Some 166,000 people representing more than 724,000,000 
adults in 24 countrles/economies ages 16-65 participated in 
the survey. 


The cognitive instruments were offered in 32 languages/ 
versions while the context questionnaire was offered in 36 
languages/versions 




(**see notes A and B in the Reader's Guide). 
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(**see notes A and B in the Reader's Guide). 
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Which countries are participating? 


Round 1 Countries Ro^nd 2 Countries 


Australia 

Italy 

Austria 

Japan 

Belgium 

Korea, Rep of 

Canada 

Netherlands 

Cyprus 

Norway 

Czech Republic 

Poland 

Denmark 

Russian Federation 

Estonia 

Slovak Republic 

Finland 

Spain 

France 

Sweden 

Germany 

United Kingdom 

Ireland 

United States 

I I Confidential and Proprietary. Copyright © 

2012 Educational Testing Service. All rights reserved. 


Chile 

Greece 

Indonesia 

Israel 

Lithuania 

New Zealand 

Singapore 

Slovenia 

Turkey 
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How Is PIAAC carried out? 

• PIAAC is administered in face-to-face interviews in 
the homes of nationally representative samples of 
adults 

• The primary mode of delivery is laptop computers 

• Adults who are unable to use a computer are 
provided with a paper and pencil assessment booklet 

• Adults who took the paper and pencil assessment 
and those with very low literacy skills also responded 
to a set of reading component measures 
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What Does PIAAC Assess? 

The main instruments in PIAAC include ... 

• Background Questionnaire 

• Reading Components 

• Literacy 

• Numeracy 

• Problem Solving in Technology Rich Environments 
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PIAAC vs lALS and ALL in the US 


lALS 

ALL 

PIAAC 



Reading 

Components 

Reading Literacy 

Reading Literacy 


Quantitative 


Numeracy 



PS in TRE 
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Selected Results 
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U.S. average literacy score (270) lower than the 
international average (273) 


[ 


Lower than in 12 countries: 

Japan, Finland, Netherlands, Australia, Sweden, 
Norway, Estonia, Flanders-Belgium, Czech 
Republic, Slovak Republic, Canada, Republic of 
Korea 

Not significantiy different than in 5 
countries: 

England and Northern Ireland- U.K., Denmark, 
Germany, Austria, Cyprus 

Higher than in 5 countries: 


Japan [ 
Finland _ 
Netherlands _ 
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Sweden _ 
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Estonia _ 
Flanders-Belgium I 
Czech Republic _ 
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International average _ 
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Denmark 
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Ireland 
France I 
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Italy I 
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U.S. average literacy score (270) lower than the 
international average (273) 


[ 


Lower than in 12 countries: 

Japan, Finland, Netherlands, Australia, Sweden, 
Norway, Estonia, Flanders-Belgium, Czech 
Republic, Slovak Republic, Canada, Republic of 
Korea 

Not significantiy different than in 5 
countries: 

England and Northern Ireland- U.K., Denmark, 
Germany, Austria, Cyprus 

Higher than in 5 countries: 
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Higher proportion of U.S. adults at the bottom 
levels of literacy 
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U.S. average numeracy score (253) lower than the 
international average (269) 


Lower than in 18 countries: 

Japan, Finland, Flanders-Belgium, Netherlands, 
Sweden, Norway, Denmark, Slovak Republic, 
Czech Republic, Austria, Estonia, Germany, 
Australia, Canada, Cyprus, Republic of Korea, 
England and Northern Ireland- U.K., Poland 

Not significantiy different than in 2 
countries: 

Ireland, France 


Higher than in 2 countries: 

Italy, Spain 



Japan 
Finland 
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A greater percentage of Whites than Blacks or 
Hispanics performed at the highest proficiency 
levels (4/5) in both literacy and numeracy 
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Only oldest U.S. adults outperformed the 
international average in literacy 
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*p < .05. U.S. average score is significantly different from PIAAC international average. 
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Literacy skills in younger and older generations 





In the U.S., a greater percentage of those who were born outside 
of the country perform at the lowest proficiency level in literacy 
and numeracy (below level 1) than those born in the country 


Yes (85) 
No (15) 

Overall 


12 

IT 


34 


25 


14 


34 



0 10 20 30 40 50 60 70 

Percent 

1 I Below level 1 Q Level 1 Q Level 2 Q Level 3 Q Level 4/5 


80 


90 


100 



Percent 

I I Below level 1 Level 1 Q Level 2 Q Level 3 | Level 4/5 



AMERICAN Institutes for Research' 







U.S. gaps in literacy scores larger than international 
average by parental education and nativity status 
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International 

average United States 


US skilled workers had the same percentage of top performers 
in literacy as the international average. Unskilled workers had a 
greater percentage of low performers than internationally. 
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Semi-skilled white-collar 
Semi-skilled blue-collar 
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In literacy, U.S. gaps larger by educational attainment 
and skiii ievei of job, but simiiar to internationai 
average by income and empioyment status 



• Avg. Education 
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Score 



U.S. average literacy score in 2012 not significantly 
different from 2003, but lower than in 1994. U.S. 
average numeracy score in 2012 lower than in 2003. 





U.S. PIAAC Findings Summary: 


■ Lower overall scores than international average in all 
subjects 

■ Higher percentages of low performers than internationally 

■ Larger gaps between less advantaged and more 
advantaged peers in literacy and numeracy, but not in 
problem solving in technology-rich environments 

■ Relatively lower performance of young adults and those with 
high school education or less 

■ Relatively higher performance of older adults in literacy and 
problem solving in technology-rich environments 
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US Release Schedule 

Released in October 2013 

■ OECD PIAAC International Report (October 8^^ 2013) 

■ NCES PIAAC National Report: "First Look" Report (October 18^^ 2013) 
Will be Released between November 2013 to February 2014 

■ OECD authored US country report (November 12, 2013) 

■ NCES PIAAC Web Portal 

■ Data Explorer (NCES) 

■ US National Technical Report (NCES) 

■ Public use data file (NCES) 

■ Restricted use data file (NCES) 

■ OECD Education and Skills Online (E & S Online) 





Time for the United States to Reskiii? What the 

Survey ofAduit Skiiis Says (An OECD authored report) 

■ Funded by the Office of Vocational and Adult 
Education, U.S. Department of Education 

■ Report will: 

■ Interpret the U.S. main findings 

■ Offer a detailed profile of low-skilled adults in the U.S. 

■ Identify policy implications and offer broad policy recommendations 
for the U.S. 

■ Be released on November 12'*^ in Washington, D.C. 

■ More detailed information regarding this event will be 
posted at www.piaacgatewav.com 
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NCES PIAAC Web Portal Content 


■ Literacy, numeracy, and problem-solving in technology-rich 
environments proficiency levels by: 

■ Country of birth 

■ Informal training 

■ Health status 

■ Profile of the employed by: age, gender, whether they are 
born in the United States, education, industry of 
employment, occupation, and earnings 

■ Profile of the unemployed by: age, gender, whether they are 
born in the United States, and education 




NCES PIAAC Data Explorer 


■ Interactive online data tool that allows users to produce 
customized PIAAC reports 

■ Data displayed and exported in a variety of formats, 
including; 

■ Tables 

■ Charts 

■ Maps 
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Additional Data Collection: U.S. National Supplement 


■ Additional household sample (3,600) for the following 
groups: 

■ unemployed adults (ages 16-65) 

■ two groups of young adults (ages 16-24 and 25-34) 

■ older adults (ages 66-74) 

■ Separate sample for 1,200 incarcerated adults (ages 16- 
74) 

■ Data collection: August 2013 and April 2014 

■ Analysis and reporting: late 2015 or early 2016 





For more information on PiAAC 


Visit AIR_PIAAC Gateway at: 

http://piaacgateway.com/ 

Contact: piaac@air.org 

Visit PIAAC at NCES: 

http://nces.ed.gov/survevs/piaac/ 
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Listening. Learning. Leading.^ 


Comparison of average literacy proficiency among young adults 

Mean literacy proficiency scores of 1 6-24 year-olds 


] Significantly above the average 
] Not significantly different from the average 
] Significantly below the average 


Mean 

Comparison country 

Countries whose mean score is NOT significantly different from the comparison country 

299 

Japan 

Finland 

297 

Finland 

Japan, Korea, Netherlands 

295 

Netherlands 

Finland, Korea 

293 

Korea 

Finland, Netherlands 

287 

Estonia 

Australia, Flanders (Belgium) 

285 

Flanders (Belgium) 

Australia, Czech Republic, Estonia, Poland, Sweden 

284 

Australia 

Czech Republic, Estonia, Germany, Poland, Sweden, Flanders (Belgium) 

283 

Sweden 

Australia, Czech Republic, Germany, Poland, Flanders (Belgium) 

281 

Poland 

Australia, Czech Republic, Germany, Sweden, Flanders (Belgium) 

281 

Czech Republic 

Australia, Austria, Canada, Denmark, Germany, Poland, Slovak Republic, Sweden, Flanders (Belgium) 

280 

Average 

Austria^ Czech Republic^ Germany^ Poland, Sweden I 

279 

Germany 

Australia, Austria, Canada, Czech Republic, Denmark, France, Norway, Poland, Slovak Republic, 
Sweden 

278 

Austria 

Canada, Czech Republic, Denmark, France, Germany, Norway, Slovak Republic 

276 

Denmark 

Austria, Canada, Czech Republic, France, Germany, Norway, Slovak Republic, United States 

276 

Slovak Republic 

Austria, Canada, Czech Republic, Denmark, France, Germany, Norway, United States 

276 

Canada 

Austria, Czech Republic, Denmark, France, Germany, Norway, Slovak Republic, United States 

275 

Norway 

Austria, Canada, Denmark, France, Germany, Ireland, Slovak Republic, United States 

275 

France 

Austria, Canada, Denmark, Germany, Norway, Slovak Republic, United States 

272 

United States 

Canada, Denmark, France, Ireland, Norway, Slovak Republic, England/N. Ireland (UK), Cyprus^ 

271 

Ireland 

Norway, United States, England/N. Ireland (UK), Cyprus^ 

267 

Cyprus^ 

Ireland, Spain, United States, England/N. Ireland (UK) 

266 

England/N. Ireland (UK) 

Ireland, Italy, Spain, United States, Cyprus^ 

264 

Spain 

Italy, England/N. Ireland (UK), Cyprus^ 

261 

Italy 

Spain, England/N. Ireland (UK) 
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Source: Survey of Adult Skills (PIAAC) (2012), Table A3.2 (L). 
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Comparison of average numeracy proficiency among young adults 

Mean numeracy proficiency scores of 16-24 year-olds 


] Significantly above the average 
] Not significantly different from the average 
] Significantly below the average 


Mean 

Comparison country 

Countries whose mean score is NOT significantly different from the comparison country 

285 

Netherlands 

Finland, japan, Korea, Flanders (Belgium) 

285 

Finland 

Japan, Korea, Netherlands, Flanders (Belgium) 

283 

Japan 

Austria, Czech Republic, Estonia, Finland, Korea, Netherlands, Slovak Republic, Sweden, 
Flanders (Belgium) 

283 

Flanders (Belgium) 

Austria, Finland, )apan, Korea, Netherlands, Slovak Republic, Sweden 

281 

Korea 

Austria, Czech Republic, Estonia, Finland, Japan, Netherlands, Slovak Republic, Sweden, 
Flanders (Belgium) 

279 

Austria 

Czech Republic, Estonia, Germany, japan, Korea, Slovak Republic, Sweden, Flanders (Belgium) 

279 

Estonia 

Austria, Czech Republic, Germany, japan, Korea, Slovak Republic, Sweden 

278 

Sweden 

Austria, Czech Republic, Estonia, Germany, japan, Korea, Slovak Republic, Flanders (Belgium) 

278 

Czech Republic 

Austria, Estonia, Germany, japan, Korea, Slovak Republic, Sweden 

278 

Slovak Republic 

Austria, Czech Republic, Estonia, Germany, japan, Korea, Sweden, Flanders (Belgium) 

275 

Germany 

Australia, Austria, Czech Republic, Denmark, Estonia, Norway, Slovak Republic, Sweden 

273 

Denmark 

Australia, Germany, Norway 

271 

Average 

Australia, Canada, Denmark, Norway, Poland 1 

271 

Norway 

Australia, Canada, Denmark, Germany, Poland 

270 

Australia 

Canada, Denmark, Germany, Norway, Poland, Cyprus^ 

269 

Poland 

Australia, Canada, Norway, Cyprus^ 

268 

Canada 

Australia, Norway, Poland, Cyprus^ 

264 

Cyprus^ 

Australia, Canada, France, Poland 

263 

France 

Cyprus’ 

258 

Ireland 

Italy, Spain, England/N. Ireland (UK) 

257 

England/N. Ireland (UK) 

Ireland, Italy, Spain 

255 

Spain 

Ireland, Italy, England/N. Ireland (UK) 

251 

Italy 

Ireland, Spain, United States, England/N. Ireland (UK) 

249 

United States 

Italy 
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Source: Survey of Adult Skills (PIAAC) (2012), Table A3.2 (N). 



Listening. Learning. Leading. 



Proficiency in problem solving in technology-rich environments among young adults 

Percentage of 16-24 year-olds scoring at each proficiency level 


□ Missing ^ Failed ICT core or had no computer experience H Below Level 1 □ Level 1 □ Level 2 ■ Level 3 


Opted out of 
the computer-based 
assessment 
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Source: Survey of Adult Skills (PIAAC) (201 2), Table A2.1 Ob. 
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The OECD 

Education and Skills Online 



Kirsch 

cational Testing Service ■ 
November 201 3 

— 7 -. 
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@ What is E & S Online 


• The OECD Education and Skills Online offers individuals, 

businesses, community based and government 
organizations a powerful new set of tools to assess their 
adult populations. 

• It is specifically designed to provide individual level data that 
is linked to PIAAC and, therefore, shares the same validity 
evidence 

• It measures both cognitive and non-cognitive skills 

• It uses a state of the art internet delivered platform that is 
available 24/7 and includes technical support 

• It is developed in 9 countries and 11 languages/versions 
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Languages / Versions 

English - US 
English - Canada 
English - UK 
English - Ireland 
Spanish - US 
Spanish - Spain 
French - Canada 
French - France 
Italian 
Czech 
Japanese 
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<^- USES and Benefits 


• state of the art internet delivered tests suitable for 
adults. 

• Individual scores in each measured domain provided 
automatically and in real time 

• Incorporates IRT, multi-stage adaptive testing and 
automated scoring 

• Available in multiple languages/versions 

• Validity evidence is linked to PIAAC measures 

• Tests cover a wide range of skills and domains 

• Individuals and organizations have flexibility to 
determine which skills to assess 
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